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FOREWORD 


This  report  (Volume  I  through  Volume  VIII)  represents  the  final  phase  of 
a  study  and  test  which  was  initiated  in  September  1964  to  explore  newly  devel¬ 
oped  techniques  and  devices  for  presenting  T.  O.  (Technical  Order)  type  in¬ 
structions  and  information.  The  eight  volumes  of  data  contain  the  result  of  a 
test  conducted  in  an  operational  environment  using  concepts  developed  during 
an  earlier  phase  under  Contract  AF  04(694)-729  and  documented  in  BSD-TR- 
65-456.  Both  the  early  phase  and  final  phases  which  were  accomplished  under 
Contract  AF  04(694)-984,  Project  1316,  "Presentation  of  Information  for  Mainte¬ 
nance  and  Operation  (PIMO)",  were  started  in  June  1966  and  completed  in 
April  1969.  This  final  report  was  submitted  in  May  1969. 

The  original  program  documentation  was  prepared  by  Mr.  C.  L.  Schaffer, 
SMTE,  in  1964.  He  subsequently  functioned  as  the  Air  Force  Program  Direc¬ 
tor  and  Chairman  of  a  Working  Group  which  monitored  all  development  through¬ 
out  the  life  of  the  project.  This  Group  was  composed  of  individuals  from  var¬ 
ious  Air  Force  commands  (AFLC,  MAC,  ATC,  ADC,  AFSC)  and  the  Army 
Command  (AMCPM,  AXMLE)  knowledgeable  in  the  various  maintenance  dis¬ 
ciplines  and  all  facets  of  the  T.  O.  system.  Capt.  Don  Tetemeyer,  the  Pro¬ 
ject  Scientist  during  the  formulative  stages  of  the  Program  was  largely  re¬ 
sponsible  for  the  basic  test  structure.  Mr.  John  Saunders  was  the  monitor 
for  all  contractual  aspects  until  his  reassignment  in  1568. 

Any  success  cne  may  attribute  to  the  project  must  be  shared  by  numerous 
individuals;  however,  major  credit  and  appreciation  are  due  General  Howell  M. 
Estes,  Jr.,  Commander  of  the  Military  Airlift  Command,  who  provided  the 
C-141A  aircraft  and  the  bases  at  Charleston,  Dover  and  Norton  for  the  opera¬ 
tional  test.  Sharing  in  the  credit  for  the  MAC  contributions  are  Lt.  Col.  Don 
Watt  and  his  staff  at  Hq.  MAC,  and  Col.  Foreman,  Col.  Henzi,  W/O  Van  Riper 
and  all  the  personnel  at  Charleston  Air  Force  Base  and  also  at  Dover  and  Norton 
who  participated  in  the  test.  The  hardships  imposed  on  their  organizations 
are  recognized,  and  we  sincerely  appreciate  the  special  efforts  put  forth  to 
overcome  all  obstacles.  The  test  could  never  have  been  conducted  without  the 
cooperation  and  competent  performance  of  these  many  individuals. 

We  are  especially  indebted  to  the  Air  Force  Human  Resources  Laboratory, 
Wright-Patterson  Air  Force  Base  for  their  financial  contributions  at  a  critical 
point  in  the  project;  and  also  to  the  Army  Materiel  Command,  who  believed  the 
test  potential  of  sufficient  magnitude  to  warrant  the  expenditure  of  their  funds. 

We  are  most  grateful  for  their  confidence  and  assistance.  It  is  most  assuredly 
the  primary  factor  that  permitted  completion  of  the  test. 

This  technical  report  has  been  reviewed  and  is  approved. 

dth. 

D.  A.  Cook,  Lt.  Col.  USAF 
Hq,  AFSC  (SCS-2) 
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ABSTRACT 


This  report  describes  the  latest  phase  in  the  program  to  develop  and 
evaluate  PIMO  (Presentation  of  Information  for  Maintenance  and  Operation); 
a  job  guide  concept  applied  to  maintenance.  Between  August  1968  and  April 
1969,  a  test  was  conducted  at  Charleston  AFB,  South  Carolina,  to  deter¬ 
mine  the  effectiveness  of  PIMO.  Three  immediate  behavioral  effects  were 
expected:  1)  reduction  in  maintenance  time,  2)  reduction  in  maintenance 
errors,  and  3)  allow  usage  of  inexperienced  technicians  with  no  significant 
penalty.  Experienced  and  inexperienced  Air  Force  technicians  performed 
maintenance  on  C-141A  aircraft  using  PIMO  Job  Guides  presented  in  audio¬ 
visual  and  booklet  modes.  Performance  was  measured  in  terms  of  time  to 
perform  and  procedural  errors.  The  performance  was  compared  with  the 
performance  on  the  same  jobs  by  a  control  group,  i.e.,  experienced  tech¬ 
nicians  performing  in  the  normal  manner.  The  following  conclusions  were 
drawn  from  the  test  results:  1)  aTer  initial  learning  trials,  both  experi¬ 
enced  and  inexperienced  technicians  using  PIMO  can  perform  error-free 
maintenance  within  the  same  time  as  experienced  technicians  performing 
in  the  normal  manner,  2)  inexperienced  technicians  perform  as  well  as 
experienced  technicians  when  both  use  PIMO,  3)  there  is  no  significant 
difference  between  audio-visual  and  booklet  modes,  4)  the  users  revealed 
an  overwhelmingly  positive  reaction  to  PIMO,  and  5)  the  performance  im¬ 
provements  provide  the  capabilities  to  significantly  improve  system  per¬ 
formance  defined  in  terms  of  departure  reliability,  time -in -maintenance, 
and  operational  readiness.  This  report  also  presents  a  description  of  the 
recommended  operational  system,  specifications  and  guidelines  for  PIMO 
format  development,  including  troubleshooting. 
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SECTION  I 
INTRODUCTION 


A.  GENERAL 


With  the  coming  of  age  of  complex  weapons  systems,  a  requirement 
arose  for  a  modern,  efficient  presentation  of  technical  information  re¬ 
lating  to  the  maintenance  of  those  systems.  Project  PIMO  (Presentation 
of  Information  for  Maintenance  and  Operations)  was  let  by  the  Air  Force 
to  Serendipity,  Inc.  to  research  and  develop  a  means  to  satisfy  this  re¬ 
quirement.  This  research  led  to  revolutionary  concepts  in  the  techniques 
associated  with  presenting  maintenance  information. 

B.  PIMO  CONCEPTS 

The  basic  concept  which  developed,  revolved  around  the  formatting  of 
maintenance  information.  This  new  formatting  quickly  led  the  investi¬ 
gators  to  consider  the  merits  of  two  separate  modes  of  presentation. 

1.  Print  Mode 

The  first  of  these  modes  was  that  of  printed  booklets  comprised  of  short, 
simple  maintenance  steps  and  informative  pictorials  for  each  step. 
Technical  procedures  were  standardized  and  the  textual  vocabulary  kept 
at  a  level  to  ensure  comprehension  by  the  user.  A  highly  stylized  pic¬ 
torial  drawing  technique  was  developed  which  controlled  the  amount  of 
information  included  only  to  that  amount  absolutely  necessary.  (Fcr 
detailed  information  concerning  the  print  mode,  see  Volume  I). 

2.  Audio-Visual  Mode 

In  addition  to  the  print  mode,  the  maintenance  information  was  organi¬ 
zed  and  presented  audio-visually.  In  this  mode,  the  pictorials  which 
were  used  in  the  booklet  were  presented  visually,  while  instructions  were 
given  by  voice  narration.  (For  detailed  information  concerning  the  audio¬ 
visual  mode,  see  Volume  I). 
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c. 


PIMO  EVALUATION 


In  order  to  make  a  valid  evaluation  of  the  relative  effectiveness  of  the 
two  presentation  modes,  a  comprehensive  test  was  required.  Once  the 
data  were  developed,  it  was  necessary  to  measure  their  relative  effective 
ness  in  a  realistic  situation.  Therefore,  with  Air  Force  approval  and 
cooperation,  a  field  test  was  constructed  and  implemented.  In  this  way, 
PIMO  concepts  were  to  be  tested  on  an  operational  aircraft  maintenance 
system  (C-141)  with  actual  maintenance  performed  following  PIMO  pro¬ 
cedures. 

D.  CONCLUSIONS 

1.  Test  Results 

The  test  results  show  that  when  PIMO  formatted  data  are  used,  appren¬ 
tices  with  and  without  specialist  training  can  perform  as  well  as  special¬ 
ists  using  conventional  Air  Force  technical  manuals.  Furthermore, 
these  apprentices,  after  repeated  exposure  to  the  same  job,  will  per¬ 
form  at  the  same  level  as  the  highly-trained  and  experienced  specialist 
performing  without  data.  This  would  tend  to  indicate  that  the  training 
for  first  term  enlistees  can  be  drastically  reduced.  They  can  be  as¬ 
signed  to  maintenance  bases  with  minimal  training  and  therefore,  be 
productive  during  most  of  their  enlistment  term.  Specifically,  a  25.  8% 
increase  in  productive  utilization  may  be  realized  if  just  the  customary 
O.  J.  T.  is  eliminated.  An  even  greater  increase  maybe  realized  if 
technical  school  time  --  G  months  --  could  be  shortened.  There  is  some 
evidence  that  the  use  of  PIMO  may  allow  for  such  a  course  truncation. 

In  the  event  that  the  technician  demonstrates  proficiency  and  willingness 
to  remain  in  the  service,  he  then  can  be  sent  to  specialist  training. 
Specialist  training  will  still  be  required  since  the  job  guides  cannot 
cover  all  activities.  However,  PIMO  Job  Guides  will  facilitate  spe¬ 
cialist  performance  when  they  are  assigned  to  new  aircraft. 
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Potential  Savings 


Our  simulation  studies  using  the  AMES  model  indicate  that  significant 
improvement  in  such  system  level  parameters  as  departure  reliability, 
time  in  maintenance  and  operational  readiness  can  be  realized  simply 
by  using  apprentices  and  non-specialists  for  what  is  currently  reserved 
for  specialists.  Again,  this  potential  will  be  realized  only  if  the  mainte¬ 
nance  managers  are  willing  to  assign  these  non-specialist  people  to  jobs 
which  can  be  performed  adequately  by  them  when  FIMO  Job  Guides  are 
available. 

In  conclusion,  then,  it  can  be  seen  that  PIMO  represents:  1)  an  impor¬ 
tant  step  forward  in  the  technique  of  data  presentation;  2)  an  improved 
capability  of  aircraft  maintenance;  and  3)  a  greater  productive  utilization 
of  Air  Force  maintenance  personnel. 

3.  Consideration  for  Implementation 

Many  of  the  results  of  the  PIMO  test  are  quite  dramatic  and  have  con¬ 
siderable  implications  for  the  Air  Force.  These  implications  can  be 
realized  only  if  the  PIMO  Job  Guide  concept  is  applied  properly,  and 
the  Air  Force  takes  advantage  of  the  potentials  of  the  PIMO  Job  Guide 
concept.  Simply  delivering  a  set  of  PIMO  Job  Guides  at  a  given  base 
will  not  necessarily  result  in  improved  performance  or  cost  savings. 

Any  organization  is  a  system.  One  portion  of  the  system  cannot  be 
changed  without  affecting  or  changing  other  subsystems  which  interact 
with  the  portion  changed.  It  would  be  erroneous  to  assume  that  a  change 
in  the  technical  data  system  can  be  accomplished  without  a  correspond¬ 
ing  change  in  training  or  maintenance  management  concepts.  This  is 
necessary  in  order  to  take  full  advantage  of  the  power  of  the  job  guides. 

In  evaluating  the  cost  effectiveness  of  the  PIMO  Job  Guides,  one  should 
also  consider  the  costs  involved  in  making  the  related  changes.  The 
related  costs  are  difficult  to  identify  at  this  time.  However,  it  must 
be  understood  that  if  the  Air  Force  is  not  willing  to  incur  the  cost  of 
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implementing  the  PIMO  Job  Guides  properly,  the  tremendous  advantages 
inherent  in  the  use  of  these  job  guides  will  be  significantly  diluted. 

PIMG  provides  the  capability  for  utilizing  both  apprentices  and  non- 
specialists  for  many  of  the  jobs  currently  reserved  for  specialists, 
but  this  added  capability  is  meaningless  if  the  maintenance  managers 
will  not  assign  the  non-specialists  and  apprentices  to  the  jobs.  In  the 
case  of  a  new  system,  the  evidence  indicates  that  PIMO  will  allow  the 
technicians  to  conduct  nearly  error-free  maintenance  from  the  outset. 

It  can  be  demonstrated  that  even  with  the  very  few  changes  suggested 
here,  significant  improvements  can  be  realized  for  new  Air  Force  sys¬ 
tems. 

4.  Data  Usage 

Many  argue  that  the  technicians  will  use  any  set  of  data  if  proper  "muscle” 
is  applied.  However,  considerable  evidence  accumulated  by  our  experi¬ 
ence  in  examining  maintenance  by  different  individuals,  at  many  dif¬ 
ferent  bases  and  for  different  commands,  indicates  otherwise.  Data 
usage  appears  to  be  an  exception  rather  tha*i  the  rule. 

Serendipity  observed  and  reported  in  an  earlier  phase  of  the  study  that 
use  of  conventional  manuals  was  less  than  5  percent.  Observations 
made  during  the  current  test  indicated  that  usage  continues  to  remain 
considerably  less  than  5  percent.  A  study  conducted  under  the  auspices 
of  the  Aeromedical  Research  Laboratory  at  Wright-Patterson  Air  Force 
Base,  tended  to  verify  these  observations. 

These  results  are  not  surprising  and  certainly  should  not  be  construed 
as  a  condemnation  of  the  conventional  manuals.  The  manuals  are  not 
designed  (nor  required  to  be  designed)  for  use  on  the  job.  It  should  not 
be  surprising,  therefore,  to  find  the  manuals  are,  indeed,  not  used  on 
the  job.  The  extent  to  which  the  manuals  are  used  off  the  job  is  not 
known.  The  FIMO  test  results  indicate  that  technical  manual  require¬ 
ments  should  now  be  changed  so  that  manuals  will  be  designed  for  use 
on  the  job,  by  both  experienced  and  inexperienced  technicians. 


The  technicians  appear  to  have  a  very  strong  biae  against  using  data 
on  the  job.  In  fact,  we  have  seen  at  least  one  occasion  where  a  crew 
chief  refused  to  use  a  technician  simply  because  he  brought  technical 
data  along  with  him.  There  appears  to  be  a  general  tendency  to  "look 
down"  upon  any  individual  who  "has  to  use  data  to  perform  the  job". 

During  the  earlier  portions  of  PIMO,  when  the  test  was  conducted  at 
three  different  bases,  we  were  operating  under  the  strict  definition 
of  non-interference.  Therefore,  we  could  only  encourage  the  tech¬ 
nicians  to  use  the  data,  i.e.,  could  not  require  them  to  use  it.  Under 
these  circumstances,  the  percentage  of  people  who  actually  used  the 
data  was  approximately  20  percent.  Although  this  figure  is  significantly 
higher  than  the  use  of  conventional  manuals,  it  was  still  not  adequate. 
Those  exposed  to  the  data  expressed  positive  acceptance,  but  still  the 
usage  was  relatively  low.  Part  of  the  low  usage  may  be  attributed  to 
the  fact  that  only  5-levels  were  assigned  to  the  jobs.  Most  of  these 
technicians  fejt  they  knew  the  job  and  therefore  did  not  need  technical 
data.  On  the  other  hand,  in  the  test  conducted  at  Charleston,  both 
usage  and  acceptance  were  significantly  higher  for  the  specialists  as 
well  as  the  apprentices.  The  difference  seemed  to  be  that  in  Charleston, 
we  spend  considerable  time  working  with  the  technicians  in  an  attempt 
to  get  them  to  overcome  whatever  bias  existed.  It  also  is  likely  that 
seeing  the  apprentices  perform  effectively  with  the  PIMO  guides  con¬ 
tributed  to  the  acceptance  by  the  specialists.  The  more  important  point 
is  that  the  specialist  did  accept  the  job  guides,  and  it  was  not  necessary 
to  apply  any  "muscle". 

It  can  be  anticipated  that  if  job  guides  are  delivered  to  a  base  heretofore 
unfamiliar  with  PIMO,  usage  will  be  higher  than  in  the  case  of  conven¬ 
tional  manuals,  but  will  not  be  sufficiently  high  to  realize  the  full  potential 
of  the  guides.  Therefore,  it  is  strongly  urged  that  a  carefully  develop¬ 
ed  program  of  orientation,  training,  and  public  relations  be  developed 
before  implementation.  Training  should  be  conducted  by  representatives 
of  the  Air  Training  Command.  Furthermore,  there  should  be  one  rep¬ 
resentative  at  each  base  who  is  thoroughly  familiar  with  the  PIMO  Job 
Guides  and  is  a  strong  supporter.  This  person  should  maintain  a  con¬ 
tinuous  educational  program  with  respect  to  PIMO  and  PIMO  Job  Guide 


usage.  In  addition,  there  should  be  a  cadre  comprised  of  one  repre¬ 
sentative  from  each  shop  for  each  shift.  This  cadre  should  be  thoroughly 
trained  on  PIMO,  so  that  they  can  answer  any  questions. 

Thus,  an  intensive  and  systematic  orientation  and  training  program  will 
have  a  significant  impact  in  realising  the  full  potential  of  the  PIMO  Job 
Guides. 


SECTION  II 


STATEMENT  OF  THE  PROBLEM 
INTRODUCTION 

The  basic  problem  addressed  by  PIMO  stems  from  an  "impasse"  re¬ 
sulting  from  the  recent  proliferation  of  complex  Air  Force  systems  and 
components.  This  proliferation  resulted  in  high  support  costs  which 
limit  the  utilization  of  the  systems. 

DEVELOPMENT  OF  THE  PROBLEM 

One  major  source  of  the  high  support  cost  is  the  current  concept  of  pro¬ 
viding  and  using  the  maintenance  manpower  capabilities.  With  the  pro¬ 
liferation  of  new  components  and  systems,  maintenance  has  become  ex¬ 
tremely  complex.  The  basic  means  of  meeting  the  increased  maintenance 
demands  is  still  one  of  using  highly  trained  and  experienced  technicians. 
However,  because  of  the  increased  complexity,  the  technicians  have  be¬ 
come  specialized.  This  specialization  limits  cross -utilization  of  per¬ 
sonnel,  thereby  increasing  the  total  number  of  personnel  required  to  sup¬ 
port  any  given  system.  The  extensive  training  given  to  the  specialists 
(both  formal  and  on-the-job)  not  only  has  increased  the  high  training  costs, 
but  also  has  limited  the  utilization  of  these  personnel  in  productive  mainte¬ 
nance  labor  to  a  relatively  small  percentage  of  the  first  enlistment  term. 
Furthermore,  since  the  technicians  have  to  receive  extensive  training 
before  they  can  be  assigned  meaningful  maintenance  jobs,  the  Air  Force 
capability  to  mobilize  quickly  during  periods  of  emergency  is  also  limited. 
In  other  words,  the  extent  to  which  we  can  increase  the  productive  mainte¬ 
nance  manpower  during  periods  of  emergency  is  limited  by  the  time  we 
have  to  take  to  train  the  new  technicians. 

When  the  system  complexity  problem  is  examined  in  terms  of  the  capa¬ 
bilities  required  of  maintenance  personnel,  it  is  interesting  to  note  that 
the  complexity  results  primarily  from  the  tremendous  increase  of  infor¬ 
mation  a  technician  has  to  bring  to  bear  on  any  given  job.  Significantly, 


this  has  not  resulted  in  any  appreciable  increase  in  motor  proficiency, 
vision,  or  hearing  capabilities;  rather,  the  maintenance  jobs  have  be¬ 
come  more  difficult  primarily  because  the  people  simply  have  to  remem¬ 
ber  more  information  relevant  to  any  given  assignment. 

If  one  accepts  the  assumption  that  information  is  the  key  to  complexity, 
it  is  not  surprising  that  the  technical  orders  {the  primary  source  of  thie 
information)  have  become  the  scapegoat  for  the  basic  problem.  However, 
it  does  not  behoove  the  Air  Force  to  restrict  their  attention  to  the  tech¬ 
nical  orders.  The  basic  problem  is  over- reliance  on  highly-trained  and 
experienced  technicians  to  accomplish  all  the  maintenance.  Examination 
of  the  required  maintenance  jobs  clearly  indicates  that  many  of  the  jobs 
can  be  accomplished  by  personnel  with  lesser  training  and  capability  than 
is  currently  designated,  if  the  information  problem  can  be  resolved.  None¬ 
theless,  this  will  not  necessarily  result  in  breaking  the  "impasse1,  un¬ 
less  training  is  adjusted  accordingly  and  personnel  with  lesser  training 
and  experience  are,  in  fact,  utilized. 

The  maintenance  situation  today  is  so  complex  that  there  is  --  and  probably 
will  continue  to  be  --  a  definite  need  for  highly-trained  and  experienced 
specialists.  But,  their  talents  should  be  applied  to  where  the  need  truly 
exists  as  in  complex  troubleshooting  situations,  supervision,  areas  re¬ 
quiring  high  motor  and  visual  proficiencies  (e.g. ,  structural  repair, 
inspection,  etc.)  --  and  to  cover  the  various  unpredictable  events  common 
within  such  complex  situations.  The  specialist  should  not  be  burdened  with 
jobs  which  can  be  handled  by  people  with  less  training  and  experience. 
Consider  the  problems  faced  by  many  bases.  Very  few  maintenance  jobs 
are  officially  designated  for  the  so-called  3-level  technicians.  Yet,  be¬ 
cause  of  personnel  turnover,  rearly  50  percent  of  the  maintenance  staff 
at  a  given  base  may  be  3-level  technicians.  The  mainten\nce  managers 
are  supposed  to  "abide  by  the  book"  and  utilize  only  the  qualified  tech- 
nicians  for  the  required  maintenance.  This  could  result  in  a  drastic  re¬ 
duction  of  aircraft  readiness;  therefore,  in  reality,  they  end  up  using 
the  3-level  technicians  simply  because  they  have  no  other  alternative. 

They  pay  a  penalty  in  maintenance  efficiency  and  reliability  in  many 
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cases,  because  the  manpower  system  is  r.ot  set  up  to  utilize  these  per¬ 
sonnel  effectively.  If  the  inexperienced  can  be  assigned  to  meaningful 
maintenance  jobs  without  a  significant  sacrifice  in  efficiency,  the  avail¬ 
able  manpower  pool  can  be  increased  from  25  to  50  percent.  This  in¬ 
crease  alone  will  be  a  great  step  forward  in  reducing  the  magnitude  of 
the  high  support  cost  problem  and  the  resultant  limitation  on  system 
utilization. 

The  above  problem  is  compounded  by  the  relatively  low  retention  of 
maintenance  technicians.  Many  of  the  individuals  who  receive  expensive 
and  extensive  specialist  training,  leave  the  Air  Force  after  a  first  term 
enlistment.  This  means  that  the  needed  manpower  resulting  from  this 
investment  is  lost  after  only  one  or  two  years  of  productivity. 

PROPOSED  SOLUTION 

—  ■  — 

The  solution  to  this  problem  can  be  to  limit  difficult  maintenance  jobs 
to  highly  trained  and  experienced  technicians.  A  key  to  resolving  the 
impasse  is  an  information  system  which  will  reduce  the  complexity  of 
the  jobs  by  eliminating  the  need  for  technicians  to  remember  all  of  the 
information  relevant  to  a  given  task  in  a  given  job.  (See  Figure  2-1. ) 
This  information  must  be  usable  on  the  job  if  we  hope  to  decrease  the 
complexity  of  the  job  through  reduction  in  memory  requirements.  This 
will  not  resolve  the  problem,  but  will  simply  provide  the  capability  to 
resolve  the  problem.  The  problem  can  be  resolved  only  if  the  capability 
is  applied,  and  maintenance  jobs  are  assigned  to  technicians  with  lesser 
training  and  experience  on  selected  jobs. 

Stated  in  another  way,  the  basic  issue  is  a  manpower  system  problem. 
The  present  manpower  system  is  not  designed  to  take  full  advantage  of 
the  available  manpower.  And,  as  with  any  other  system  problem,  it 
cannot  be  resolved  by  changing  only  one  portion  of  the  system,  e.g. , 
information  subsystem.  The  interfacing  portions  must  be  changed  to 
assure  that  the  system  is  improved  indeed. 


Figure  2-1.  Memory  requirements,  training  vs 
job  guides. 


SECTION  III 
HISTORY  OF  PIMO 
A.  PROBLEM  DEFINITION 

In  1964,  the  United  States  Air  Force  began  a  concentrated  effort  to 
analyze  mounting  problems  associated  with  the  newer,  more  complex 
and  more  sophisticated  military  aircraft  systems.  One  area  of  in¬ 
vestigation  which  seemed  potentially  able  to  yield  significant  returns 
was  that  of  aircraft  maintenance.  It  was  discovered  that  the  standard 
Air  Force  technical  orders,  the  vehicle  used  to  present  information  to 
the  maintenance  technicians,  had  gone  without  significant  change  for  ihe 
last  thirty  years.  These  technical  orders  were  originally  designed  for 
the  experienced  specialist,  and  justifiably  so.  However,  the  rapid  in¬ 
crease  in  techniques  for  information  presentation  seemed  to  indicate 
that  these  new  techniques  could  be  applied  to  the  modernization  of  the 
technical  orders  with  some  real  benefits.  For  example:  technical  or¬ 
ders  denied  the  maintenance  base  the  ability  to  use  apprentice  tech¬ 
nicians.  It  seemed  to  follow  logically,  that  through  changes  in  tech 
data,  lower  skill  level  personnel  could  be  used,  thereby  freeing  the  ex¬ 
perienced  technician  for  more  complex  jobs  and  increasing  the  popu¬ 
lation  of  competent  maintenance  technicians. 

While  the  problem  of  technical  data  modernization  was  not  the  only  one 
investigated  by  the  Air  Force  at  this  time,  it  is  from  that  area  specifi¬ 
cally  that  this  project  grew.  Therefore,  in  that  same  year.  Lieutenant 
General  W.  A.  Davis,  Vice  Commander,  AFSC,  initiated  a  working 
group  with  the  specific  assignment  to  study  ways  in  which  the  technical 
order  system  could  be  improved.  This  group  was  composed  of  members 
from  all  major  Air  Force  agencies,  plus  members  from  the  other  ser¬ 
vices.  The  group  was  charged  with  the  responsibility  to  1)  develop  a 
work  statement,  2)  issue  an  RFP,  and  3)  monitor  the  subsequent  con¬ 
tract. 


In  1965,  two  contracts  were  let  (see  Figure  3-1).  The  first,  issued 
to  Serendipity,  Inc. ,  was  to  study  the  effects  of  improved  job  guides  on 
the  UH-1F  Helicopter  system.  The  second  was  let  to  R.  M.  Parsons 
Company  to  perform  a  similar  study  on  the  BUIC-416M  (Back  Up  Inter¬ 
ceptor  Control)  system.  The  conclusions  drawn  from  these  studies 
emphasized  the  need  for  the  application  of  job  guide  concepts,  and  for 
the  generation  of  improved  techniques  of  information  formatting. 
Serendipity,  through  both  basic  and  applied  research,  developed  what 
seemed  to  be  a  workable,  new  formatting  technique.  Moreover,  there 
was  some  indication  that  perhaps  two  modes  of  information  presentation 
warranted  testing.  The  first  --  print  mode  --  was  comprised  of  the 
newly  formatted  maintenance  instructions.  This  new  print  format  stress¬ 
ed  simplicity  in  text  and  illustration.  The  second,  a  somewhat  more 
sophisticated  mode,  employed  audio-visual  presentation.  It  was  in¬ 
tended  that  this  technique  be  capable  of  presenting  maintenance  in¬ 
structions  on-board  aircraft  by  displaying  the  pictorial  (as  developed 
for  the  print  mode)  visually,  while  presenting  the  textual  material 
with  voice  narration. 

PILOT  TESTING 

The  new  concepts,  under  the  acronym  PIMO  ^Presentation  of  Information 
for  Maintenance  and  Operations),  were  laboratory  tested  during  Phase 
1A  of  the  PIMO  Program.  This  test  employed  three  modes  of  presenta¬ 
tion:  audio-visual,  printed  manuals  (containing  instructions  identical 
to  the  audio-visual),  and  standard  USAF  technical  orders.  The  test 
was  conducted  in  a  shop  using  selected  components  from  the  UH-1F 
Helicopter  system. 

The  upshot  of  this  study  indicated  that  PIMO  was  indeed  effective  in  re¬ 
ducing  maintenance  times  and  errors,  and  that  the  concept  warranted 
further  study.  Results  of  the  Phase  1A  test  were  presented  to  the  Air 
Staff,  who  then  decided  to  initiate  Phase  IB.  This  phase,  again  awarded 
to  Serendipity  for  study,  revolved  around  a  small  feasibility  study  in 
which  PIMO  concepts  were  applied  to  the  C-141A  maintenance  program. 
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The  conclusions  of  Phase  IB  indicated  that  a  significant  savings  in 
manpower  and  money  could  be  experienced  if  the  PIMO  concepts  were 
adopted.  But  a  final  comprehensive  and  exhaustive  field  test  was  called 
for  in  order  to  gain  a  substantial  validation  of  the  general  and  specific 
effects  of  PIMO  on  a  cystem-wide  basis.  That  study  was  conducted 
during  1967-69  at  Charleston,  Dover  and  Norton  Air  Force  Bases.  The 
results  of  this  final  study  demonstrated  conclusively  that  significant 
improvements  in  over-all  system  effectiveness  can  be  realized  through 
the  utilization  of  PIMO  Job  Guides, 


SECTION  IV 


PIMO  FIELD  TEST  DESIGN  AND  IMPLEMENTATION 

A.  INTRODUCTION 

Since  the  field  tests  Associated  with  the  earlier  phases  of  the  PIMO 
program  have  been  reported  elsewhere,  and  since  the  last  field  test 
is  the  natural  extension  and  culmination  of  all  previous  effort,  this 
section  will  be  devoted  to' the  description  of  that  last  test. 

The  subject  test  was  conducted  at  Charleston  Air  Force  Base,  South 
Carolina,  between  August  19S8  and  March  1969.  Two  full-time  Seren¬ 
dipity,  Inc.  representatives  were  stationed  at  Charleston  throughout 
this  period,  and  were  supplemented  as  required  by  other  Serendipity 
staff  members. 

B.  FIELD  TEST  DESIGN 
1.  Objectives 

The  goal  of  the  PIMO  field  test  was  to  permit  an  objective  assessment 
of  the  PIMO  concept  on  aircraft  maintenance  and  subsequent  aircraft 
system  effectiveness.  Earlier  studies  concerned  with  the  use  and  ef¬ 
fectiveness  of  the  PIMO  concept  --  when  applied  to  helicopter  mainte¬ 
nance  --  led  to  the  generation  of  five  specific  hypotheses.  These  hy¬ 
potheses  stated  that,  through  the  use  ol'  PIMO  formatted  data: 

a.  A  decrease  in  maintenance  time  of  at  least  10  percent 
would  be  experienced. 

b.  PIMO  data  presented  via  audio-visual  mode  would  result 
in  an  additional  10  percent  reduction  in  maintenance  man-hours. 

c.  A  45  percent  reduction  of  maintenance  errors  would  be 
possible  when  technicians  use  PIMO  Job  Guides. 
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d.  For  80  percent  of  the  maintenance  tasks  which  might  occur, 
a  3-level  technician  with  PIMO  could  perform  as  well  as  a  5-level  tech¬ 
nician  with  standard  USAF  technical  orders. 

e.  A  significant  reduction  in  system-wide  spares  consumption 
would  result. 

Stated  symbolically,  these  working  hypotheses,  together  with  their  cor¬ 
responding  null  hypotheses,  appear  in  Figure  4-1.  More  generally, 
there  were  three  immediate  behavioral  effects  which  were  expected. 

Stated  simply,  these  effects  were: 

1.  Reduction  in  maintenance  time. 

2.  Reduction  in  maintenance  errors. 

3.  Ability  of  apprentice  technicians  to  perform  specialized 
maintenance  with  the  use  of  PIMO. 

The  data  taken  during  the  main  testing  effort,  in  addition  to  being  sub¬ 
jected  to  standard  statistical  analysis  techniques,  were  used  to  provide 
the  base  for  an  exercise  of  the  AMES  Model.  This  model,  described  in 
detail  in  Section  VII,  provides  for  an  evaluation  of  the  effect  of  improve¬ 
ment  in  aircraft  maintenance  on  over-all  system  effectiveness,  e.g., 
flying  hours,  sorties. 

In  addition,  a  separate  study  --  accomplished  concurrently  wif-h  the 
main  study  --  was  made  to  evaluate  the  effects  of  a  set  of  Troubleshooting 
Aids  (TSA's)  which  incorporated  a  series  of  Maintenance  Dependency 
Charts  (MDC's)  for  fault  isolation. 


Lastly,  an  assessment  of  user  attitude  and  acceptance  of  PIMO  was 
made  through  the  use  of  an  attitude  questionnaire. 
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Figure  4-1.  Specific  Hypotheses  for  PIMO  Field  Test 


2.  Experimental  Design 

a.  General  Characteristics 

The  experimental  paradigm  selected  for  the  basic  study  was  PxQx 
R  factorial  design  (see  Figure  4-2).  In  this  way,  it  would  be  possible 
to  employ  the  analysis -of-variance  model  for  data  reduction  and 
interpretation.  It  is  generally  accepted  that  the  analysis  of  variance 
method  --  especially  suited  for  factorial  designs  --  provides  a  high 
level  of  confidence  in  subsequent  inferences,  allows  for  an  inspection 
of  interaction  effects,  and  calls  for  the  greatest  economy  of  calcula¬ 
tions. 

For  the  analysis  of  the  TSA  evaluation  study,  student's  t  was  employed 
rather  than  analysis  of  variance,  since  the  paradigm  would  not  contain 
as  many  dependent  variables. 

p 

For  analysis  of  the  questionnaire  data,  X *  was  originally  selected, 
but  because  the  data  would  later  show  such  an  overwhelming  and 
unambiguous  positive  response,  only  descriptive  statistics  are  reported. 1 

Since  field  tests  are  rarely  as  closely  controlled  as  those  conducted 
under  laboratory  circumstances,  it  was  decided  to  accept  the.  05  level 
of  confidence. 

b. -  Specific  Characteristics 

1)  Independent  Variables  --  the  two  main  independent 
variables  to  be  considered  were  presentation  of  PIMO 
formatted  data,  a)  via  booklets,  or  b)  via  audio-visual 
techniques. 

In  addition,  four  maintenance  activities  were  selected 
for  investigation. 


For  a  large  number  of  questions,  for  example,  the  distribution  of 
responses  would  be,  or  would  approximate,  100%:0%. 
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a)  Remove  -  removal  of  a  bad  component  from  the  system. 

b)  Install  -  installation  of  a  good  component  into  a  system. 

c)  Adjust  -  adjusting  of  a  component  in  a  system  that  is 
out  of  tolerance. 

d)  Operational  Checkout  -  testing  of  a  system  to  verify 
that  it  is  within  tolerance. 

Since  for  each  of  those  maintenance  functions  there  existed 
separate  activities  which  varied  widely  with  time,  a 
decision  was  made  arbitrarily  to  divide  the  activities 
associated  under  each  function  into  three  major  groupings-- 
long,  medium  and  short  —  based  on  the  number  of  steps 
involved  to  complete  any  given  activity. 

For  the  concurrent  study  to  determine  the  effectiveness  of 
the  Troubleshooting  Aids,  the  independent  variable  was  the 
use  of  the  TSA  manuals  balanced  against  troubleshooting 
activities  performed  under  the  on-going  practices  found 
at  the  base. 

2)  Dependent  Variables  --  The  first  dependent  variable  was 
the  total  time  taken  to  complete  a  maintenance  action. 

This  was  labeled  Time-in-Function  and  abbreviated  TIF. 
This  time  was  recorded  using  minutes  as  the  unit  of 
measure.  Performance  error  rate  was  the  second 
dependent  variable  taken,  with  the  ground  rule  that  an 
error  was  committed  when  a  subject  omitted  a  task, 
performed  the  wrong  task,  or  had  to  repeat  the  task  or 
procedure. 

3)  Subjects  --  Two  groups  of  subjects  were  required.  The 
first  group  was  drawn  somewhat  at  random  from  the 
population  of  experienced  skilled  technicians  qualified 

to  perform  maintenance.  The  second  group  was  selected 


from  the  lower  skill-level  technicians  and  members  of 
the  Operational  Maintenance  Squadron  who  were  not 
qualified  to  perform  maintenance  on  the  C-141A's  (see 
Figure  4-3).  Each  member  of  the  test  group  served  as 
his  own  control,  in  that  ail  subjects  employed  both  tech  ¬ 
niques  of  data  presentation.  A  split-half  technique  was 
used,  whereby  half  of  the  members  of  each  group  were 
exposed  to  one  mode  of  data  presentation  (e.g. ,  booklets), 
while  the  other  half  of  each  group  was  exposed  to  the 
alternate  mode  of  data  presentation(e.  g. ,  audio-visual). 
Midway  through  the  test,  the  data  presentation  modes 
were  reversed.  In  this  way,  any  effects  due  to  the  order 
of  presentation  mode  were  counteracted.  Each  group  of 
subjects  --  experienced  and  inexperienced  --  was 
composed  of  18  men  of  approximately  similar  background, 
time  in  service,  age,  and  level  of  experience.  It  was  not 
expacted  that  an  exact  match  could  be  made,  since  the 
selection  of  each  individual  rested  in  the  hands  of  the  Air 
Force,  and  that  selection  waa  predicated  in  part  on  the 
individual's  availability.  Nonetheless,  demographic  data 
on  each  group  indicated  that  there  was  a  high  degree  of 
within-group  homogeneity.  Another  requirement  estab¬ 
lished  was  that  all  subject  participants  could  be  made 
available  throughout  the  field  tests.  That  is,  no  losses 
in  subjects  would  be  encountered  due  to  previously  planned 
leaves  or  discharge  from  the  service.  This  requirement 
placed  further  limitations  on  acquiring  a  precisely  matched 
group. 

While  a  control  group  was  used,  no  personnel  selection 
was  necessary  due  to  the  nature  of  taking  control  group 
data.  It  was  decided  that  the  control  data  would  be  de¬ 
rived  from  maintenance  times  --  for  the  tasks  under  in  - 
vestigation  with  PIMO  formats  --  by  specialists  using 
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standard  on-going  procedures  in  effect  at  the  base.  In 

f 

this  way,  it  was  felt  that  a  realistic  comparison  could  be 
made  later  of  the  effects  of  the  PIMO  format.  Control 
data  were  taken  by  trained  PIMO  team  observers  accom¬ 
panying  a  regular  base  maintenance  spiecialirt  during  the 
course  of  his  maintenance  activities.  1 

.  f 

4)  Observers  --  Data  collectors  (observers)  were  selected 
jointly  by  the  Air  Force  and  Serendipity  from  the  experi¬ 
enced  maintenance  specialist  population  at  Charleston 

Air  Force  Base.  It  was  necessary  i/hat  our  observers: 

» 

1)  be  knowledgeable  in  standard  Air;  Force  maintenance 
operations,  2)  be  specialists  in  at  least  one  skill,  and 
3)  hold  a  rank  no  less  than  sergeant.  The  need  for  these 
observers  to  be  rated  specialists,  was  so  that  an  inex¬ 
perienced  subject  performing  on-aircraft  maintenance 
would  be  restrained  from  committing  errors  that  would 

*  i 

lead  to  immediate  danger  to  himself,  or  subsequent 
catastrophic  failure  of  the  aircraft.  In  addition.  Air  Force 
policy  requires  that  all  on-aircraft  maintenance  be 
accomplished,  or  approved  by,  a. rated  specialist.  Thus, 
maintenance  actions  performed  by  the  inexperienced 
technician  could  be  subsequently  "signed-off",  since  it 
had  been  performed  under  the  observation  of  the  skilled 
technician/  observer. 

The  observers  were  given  a  5 -day  training  class  by  a 
senior  Serendipity  staff.  This  course  warned  against  the 
more  obvious  pitfalls  of  experimental  data  taking  which 
must  be  avoided;  and  it  reinforced  the  position  that  the 
observers  were  never  to  make  a  contribution  to  a  main¬ 
tenance  action  exceptin  those  cases  where,  in  their 
judgment,  the  observed  subject  was  about  to  commit  an 
error  which  would  place  himself  or  the  aircraft  in  danger. 


.  - 
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5)  Equipment  --  The  original  test  plan  was  to  present  the 
maintenance  data  in  the  audio-visual  mode  with  an  auto¬ 
mated  suite  of  equipment.  This  equipment  suite  was  to 
consist  of  two  major  classes.  One  class  was  to  be  that 
of  a  local  visual  display  device  and  handset  for  addressing 
the  required  information  frame.  The  second  class  --  to 
be  contained  in  a  trailer  located  near  the  flight  line  --  was 
to  be  comprised  of  audio  equipment  driven  by  a  PDP-8 
computer. 

This  automated  audio-visual  presentation  equipment  was 
designed  to  operate  in  the  following  way.  First,  the  sub¬ 
ject  maintenance  technician  would  call  up  an  index  of 
addresses  to  locate  the  desired  maintenance  information. 
Next,  this  address  would  be  inserted  into  the  automated 
system  via  the  handset.  The  trailer  audio  equipment 
computer  would  begin  search  of  a  master  audio  track  to 
locate  the  activity  called  for.  Once  located,  the  audio 
information- would  transfer  to  one  of  ten  satellite  recorders, 
where  it  was  made  available  to  the  maintenance  techni¬ 
cian.  At  the  same  time,  the  visual  information  data  base 
would  be  searched  for  the  addressed  frame  locally,  at 
the  visual  device.  It  became  obvious  when  the  visual 
information  was  ready,  since  that  frame  would  be  projected 
to,  pr  identified  by,  the  technician.  A  ready  light  on  the 
handset  indicated  that  the  audio  information  was  also 
ready  to  begin.  This  system  was  so  designed  that  once  it 
was  accessed,  the  audio  and  visual  information  were 
locked  in  synchrony.  This  system  would  give  the  techni¬ 
cian  the  potential  capability  of  complete  random  access  to 
any  information  frame,  together  with  an  unlimited  number 
of  repeal  audio  bursts,  as  desired  --  again  on  a  random 
basis. 
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A  requirement  analysis  study  was  conducted  first  in  order 
to  determine  the  operational  characteristics  necessary 
for  the  audio-visual  system.  These  requirements  were 
translated  into  engineering  de-sign  specifications,  and 
subcontracts  were  let  for  procurement  of  the  actual  equip¬ 
ment.  The  basic  audio  equipment  was  manufactured  and 
assembled  by  Babcock  Electronics  Corporation  of  Costa 
Mesa,  California.  The  tape  drives  for  the  audio  equip¬ 
ment  were  manufactured  by  Telect.ro  Corporation  of 
New  York.  The  visual  device  selected  was  the  Republic 
Aviation  Corporation  -  Fairchild -Hiller  Division's 
Micro-Vue,  Mark  IDb.  The  Micro-Vue  unit  could  store 
a  . maximum  of  ten  microfische;  each  fische  was  capable 
of  reproducing  a  page  matrix  69  x  69  frames.  These 
fische  measured  approximately  4x5  inches. 

The  audio  equipment  was  to  be  controlled  exclusively  by 
the  PDP-8  computer.  That  computer  was  to  be  augmented 
by  a  buffer  unit,  comprised  of  Digital  Electronic  Corpora¬ 
tion  modules,  but  designed  by  Serendipity  staff  and  assem¬ 
bled  on  subcontract.  A  master  audio  recording  console 
was  designed  by  Serendipity,  fabricated  by  Babcock 
Electronics,  and  installed  at  Serendipity's  Western 
Division  offices  at  Chatsworth,  California.  This  recording 
console  automatically  recorded  the  necessary  tone  codes 
for  proper  stop-start,  address  identification.  Miscella¬ 
neous  units  of  GFE  electronics  completed  the  equipment 
suite,  e.g. ,  power  supplies  and  intercoms.  A  professional 
announcer  was  employed  by  Serendipity  to  record  the 
entire  audio  narration.  It  was  found  that  approximately 
80  hours  were  necessary  to  contain  the  entire  maintek.  ance 
data  base. 

According  to  the  requirements  established  by  Serendipity's 
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engineers.  Charleston  Air  Force  Base  civil  engineers 
designed  and  constructed  cable-runs  from  the  audio 
equipment  trailer  to  six  maintenance  locations  on  the  base. 
This  provided  a  hard -wire  link  between  the  Micro-Vue 
used  locally  at  the  aircraft,  and  the  trailer  which  then 
supplied  both  power  and  a  signal  /.ath  for  address  codes 
and  subsequent  narration. 

Five  weeks  before  the  data  taking  phase  of  the  field  test 
was  to  begin,  all  patents,  plans,  spares  and  technical 
personnel  associated  with  the  Micro-Vue  were  acquired 
from  Fairchild -Hiller  by  Microform  Data  Corporation  of 
Palo  Alto,  California.  Conditions  associated  With  that 
transfer  obviated  production  of  updated  microfische 
necessary  for  the  accomplishment,  of  the  field  test  at 
Charleston.  Therefore,  it  became  necessary  to  substitute 
other  audio-visual  equipment  for  that  which  was  originally 
intended  to  be  used. 1 

A  search  of  the  available  audio-visual  devices  disclosed 
a  unit  which  represented  the  least  compromise  from  the 
original  concept.  This  unit  was  manufactured  byAudiscan 
Incorporated  of  Bellevue,  Washington  (see  Figure  4-4). 
While  the  Audiscan  had  neither  complete  random  access 
nor  audio  repeatability,  it  was  felt  that  through  the  judicious 
organization  of  the  data  and  through  the  use  of  some  manual 
narration  techniques,  the  disadvantages,  if  any,  would  be 
greatly  lessened.  The  Audiscan  device  weighed  12-1/2 
lbs.,  was  slightly  more  than  3  cu.  ft.  and  used  a  standard 
110  volt  power  source.  Information  for  display  was  stored 
in  2-sided  cartridges;  one  side  contained  the  visual  infor¬ 
mation  on  single-frame  16  mm.  motion  picture  film,  and 


Concurrently,  we  encountered  considerable  difficulties  in  getting  the 
audio  equipment  operational.  Although  these  difficulties  were  being 
overcome,  it  was  at  the  expense  of  manhours  which  would  be  wasted 
if  the  Micro-Vue  could  not  be  used. 
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the  other  side  contained  the  audio  information  on  standard 
1/4  inch  recording  tape.  The  capacity  of  each  cartridge 
was  limited  to  225  frames  of  visual,  and  30  minutes  of 
continuous  audio. 

The  Audiscan  electronic  logic  circuits  were  so  designed 
that  the  information  for  display  was  programmed  to 
sequence  upon  the  request  of  the  operator.  Inasmuch  as 
there  was  no  capability  to  skip-step  forward  (other  than 
in  real  time),  it  was  necessary  to  place  no  more  than  one 
maintenance  activity  in  a  cartridge.  While  this  provided 
a  semi -random  access  the  first  frame  for  each  activity 
was  randomly  accessible  --it  meant  that  the  maximum 
capability  for  each  cartridge  would  rarely  be  realized. 

As  a  consequence,  146  cartridges  were  necessary  to 
contain  the  data  base.  And,  since  two  complete  sets  of 
data  were  thought  to  be  necessary,  a  total  of  292  cartridges 
were  used.  Because  the  Audiscan  system  was  completely 
self-contained,  requirement  for  the  trailer  audio  no  longer 
existed;  therefore,  a  decision  was  made  to  close  down  the 
trailer.  The  narration  heretofore  recorded  for  the 
trailer  system,  was  then  transferred  to  the  appropriate 
Audiscan  cartridges.  The  visual  material  (pictorial)  was 
photographed  and  reproduced  on  16  mm.  motion  picture 
film  as  necessary  for  inclusion  in  the  Audiscan  cartridges. 
Each  cartridge  was  separately  and  exhaustively  inspected 
to  ensure  that  the  corresponding  visual  and  audio  material 
were  loaded  in  the  same  cartridge  and  in  synchrony. 
Arrangements  were  made  through  the  base  to  provide  a 
source  of  110  volt,  60  cycle,  a.c.  power  at  each  aircraft 
while  maintenance  was  being  performed  by  a  PIMO  subject 
using  the  Audiscan. 

The  visual-only  material  (booklets)  (see  Figure  4-5)  re¬ 
quired  just  that  they  be  updated  to  include  the  latest 
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equipment  modifications.  The  necessary  change  pages 
were  then  inserted  into  the  series  of  volumes. 

Each  observer  was  supplied  with  a  watch  suitable  for 
timing  the  maintenance  activities  under  observation, 
and  a  clipboard  with  data  collection  sheets.  Miscellaneous 
equipment  such  as  pencils,  erasers,  etc.  was  also 
provided.  Electronic  test  equipment,  tools  and  other 
maintenance  hardware  were  obtained  from  the  particular 
maintenance  shops  as  needed. 

A  ready  room  was  provided  in  a  maintenance  dock  for  the 
subjects,  observers  and  dispatcher.  Telephone  service 
was  also  provided  for  the  ready  room,  so  that  contact 
could  be  made  with  the  maintenance  shops  and  the  base 
maintenance  dispatch  office. 

6)  Procedure  -  General  --  Subsequent  to  the  selection  of 
both  observer  and  subject  personnel,  and  prior  to  the 
onset  of  data  collection,  a  training  course  was  given  in 
order  to:  1)  orient  the  team  to  the  methods  and  goals  of 
the  study,  2)  caution  them  against  violating  certain  primary 
and  basic  principles  of  experimentation,  and  3)  create 
a  rapport  between  Air  Force  and  Serendipity  personnel. 

At  that  time,  instruction  was  given  in  the  use  of  the 
Audiscan,  PIMO  booklets  and  the  Troubleshooting  Aids. 

As  noted  earlier,  there  were  two  subject  groups.  The 
first  group,  called  Group  A ,  included  experienced, 
qualified  maintenance  specialists.  The  second  group, 
called  Group  B,  contained  apprentice  or  inexperienced 
technicians.  Half  of  the  members  from  each  group  began 
the  study  using  the  audio-visual  mode,  and  the  other  half 
of  the  group  began  the  study  using  the  print  mode.  Half¬ 
way  through  the  test,  these  groups  were  inverted  so  that 


each  group  would  experience  both  presentation  modes, 
and  ordering  effects  would  be  somewhat  controlled. 

When  an  assignment  for  aircraft  maintenance  came  in,  the 
subject  and  his  observer,  together  with  the  necessary 
data  and  tools,  went  to  the  aircraft.  (See  Figure  4-6). 

Once  there,  with  all  tools  and  maintenance  data  made 
ready,  the  activity  was  begun.  The  observer,  a  qualified 
specialist,  recorded  the  elapsed  time.  Upon  completion 
of  the  maintenance  activity,  the  elapsed  time  was  noted 
and  the  proper  aircraft  maintenance  documentation  was 
completed.  In  those  cases  where  the  subject  was  inex¬ 
perienced,  the  observer  certified  the  maintenance  activity 
as  to  its  completeness  and  accuracy.  The  subject  and 
observer  then  redurned  to  the  ready  room,  where  the 
observer  reported  the  Time-in-Function  (TIF)  and  errors 
to  the  watch  supervisor.  At  this  point,  both  subject  and 
observer  were  considered  to  be  available  for  any  sub¬ 
sequent,  appropriate  maintenance  assignments. 

7)  Procedure  -  Specific  --  Upon  notification  that  an  aircraft 
required  maintenance.  Serendipity  personnel  decided 
which  data  presentation  mode  would  be  used  and  which 
test  group,  A  or  B,  would  perform  the  maintenance. 

This  decision  was  based  upon  a  prearranged  schedule  that 
was  intended  to  guarantee  that  the  correct  number  of  data 
points  would  be  taken  for  each  cell.  This  schedule  require¬ 
ment  matrix  was  updated  on  a  periodic  basis  as  the  main¬ 
tenance  data  were  accumulated  in  order  to  ensure  that  no 
more  sample  points  were  taken  for  any  given  activity 
than  was  actually  required. 

Appropriate  PIMO  booklet  or  audio-visual  device  and 
necessary  cartridge  was  taken  from  the  library  main¬ 
tained  in  the  PIMO  work -center  ready  room.  Once  at  the 
work  site,  the  observer  prepared  the  necessary  equipment 
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if  applicable,  and  prior  to  beginning  maintenance,  the 
observer  noted  all  necessary  data  for  identifying  the 
maintenance  action  to  be  initiated,  e.g.,  job  control 
number,  worker's  name  and  number,  aircraft  serial 
number,  flying  hours,  etc.  Once  these  data  were  recorded, 
the  observer  noted  the  time  maintenance  actually  began. 

If  at  any  tims  the  maintenance  was  stopped,  either  because 
of  completion  or  delay,  the  observer  again  recorded  the 
stop  time.  In  these  instances  where  a  delay  was  experi¬ 
enced,  a  continuation  of  maintenance  was  recorded  and  a 
new  start/stop  time  entered.  The  observer  not  only  re¬ 
corded  the  time  data  but  also  witnessed  errors  committed 
during  the  execution  of  maintenance.  These  errors  were 
coded  and  recorded  on  an  error  sheet  attached  to  a 
Serendipity  designed  PIMO  Data  Record  (PDR).  On  this 
card,  the  following  data  were  recorded:  (See  Figure  4-7). 

a)  Air  Force  job  control  number  --  necessary  for 
relating  the  maintenance  action  to  Air  Force 
records. 

b)  Aircraft  serial  number  --  necessary  for  perfor¬ 
ming  progressive  on-going  analyses  of  aircraft 
status . 

c)  Technician  performing  maintenance  --  necessary 
for  analyses  at  individual  level. 

di  Aircraft  system  work  unit  code  and  function  -- 
all  necessary  for  data  organization. 

It  should  be  noted  here  that  data  collection  entailed 
monitoring  of  two  separate  parameters,  as  implied  above 
--  the  elapsed  time  to  perform  the  maintenance,  called 
Time-in-Function,  and  error  rate.  The  time  data  in¬ 
cluded  only  that  time  expended  against  actual  maintenance. 
Travel  time,  set-up  time,  put-away  time,  wait-time  for 
parts  acquisition,  etc.,  were  excluded  from  the  Time-in- 
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Function  data.  Error  data  collection  was  somewhat  more 
complex.  In  order  for  the  data  collector  or  observer  to 
identify  errors,  he  must  first  follow  the  instructions 
given  by  the  job  guide.  In  those  instances  whfere  a  mem¬ 
ber  of  the  control  group  was  performing  maintenance  -- 
that  is,  performing  maintenance  without  the  use  of  techni¬ 
cal  orders  or  job  guides  —  the  observer  had  to  be  particu¬ 
larly  attentive  in  order  to  identify  procedural  errors. 

In  those  cases  where  the  subject  was  a  member  of  Group 
B  --  the  unskilled  group  --  the  observer  assigned  was  a 
skilled  technician  who  normally  would  work  on  the  affected 
system.  At  no  time  was  this  observer  to  aid  the  unskilled 
technician  on  maintenance  performance  except,  as  noted 
above,  when  an  act  would  lead  to  immediate  harm  to 
either  the  subject  or  the  aircraft. 

Under  normal  conditions,  observers.  Serendipity  person¬ 
nel,  and  the  unskilled  group  of  technicians  would  reside 
in  the  work-center  ready  room.  The  skilled  group  subject 
personnel  normally  remained  active  in  their  respective 
shops  until  called  upon  by  the  PIMO  work  center. 

8)  Data  Requirements  --  The  reader's  attention  is  directed 
back  to  Figure  4-2  which  graphically  illustrates  the  test 
design  model.  Note  that  this  test  design  model  has  a 
total  of  60  cells.  Experimental  design  led  to  the  need 
for  20  sample  points  per  cell  --  for  a  total  of  1,  200 
sample  points  in  all.  In  addition,  20  sample  points  per 
cell  were  required  for  the  TSA  effectiveness  study. 

Upon  examination  of  the  historical  maintenance  actions 
data  associated  with  C-141A  aircraft  assigned  to 
Charleston  Air  Force  Base,  it  was  predicted  that  a 
maximum  of  4-1/2  months  would  be  required  to  obtain 


these  sample  points.  This  was  based  on  separating  the 
experimental  subject  teams  in  two  shifts,  .5  days  a  week. 
Since  the  schedule  would  extend  through  the  Thanksgiving, 
Christmas  and  New  Year  holidays,  an  allowance  was  made 
for  possible  reduction  in  the  data  taking  rate  during  that 
time.  A  master  data -taking/personnel -assignment  matrix 
was  generated  as  a  management  tool  in  order  that  data 
would  be  taken  for  all  cells  without  the  expenditure  of 
unnecessary  resources. 

Although  the  field  test  data -taking  phase  schedule  had  been 
predicated  on  historical  data  as  mentioned  above,  experi¬ 
ence  in  the  field  quickly  demonstrated  that  a  significantly 
lesser  failure  rate  was  in  effect.  There  were  no  obvious 
clues  as  to  why  this  was  so.  *  Nonetheless,  it  became 
obvious  very  early  that  if  the  lesser  failure  rate  were  to 
continue,  it  would  be  necessary  to  extend  the  data  col¬ 
lecting  activities  for  an  additional  12  or  14  months.  Since 
almost  all  C-141's  were  candidate  aircraft  for  this  study, 
instead  of  the  few  originally  assigned,  the  problem  seemed 
severe  indeed.  The  solution  was  suggested  by  Lt.  Col. 
Henzi,  Deputy  Chief  of  Maintenance.  He  suggested  that 
we  perform  the  special  maintenance  activities  on  NORS 
and  IRAN  aircraft.  This  allowed  us  to  accelerate 
our  data  taking  rate  to  the  point  where  as  it  later  developed, 
we  were  able  to  find  time  to  perform  a  heretofore  unan¬ 
ticipated  special  test  at  the  end  of  the  main  study. 

It  should  be  made  clear,  that  the  use  of  the  term,  "special" 
does  not  mean  that  faulted  components  or  systems  y,c~s 
inserted  into  the  aircraft  for  subsequent  maintenance,  to 
be  accomplished  by  PIMO  subjects.  In  fact,  this  was  not 
required,  because  the  maintenance  specialists  acting  as 
subjects  in  the  main  study  were  not  required  to  trouble - 

In  fact,  no  attempt  will  be  made  here  to  explain  this  phenomena. 
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shoot.  Consequently,  the  condition  of  the  maintained 
item  was,  in  a  sense,  irrelevant. 

Of  course,  this  procedure  was  not  possible  for  the  TSA 
effectiveness  study;  thus,  all  troubleshooting  data  were 
collected  on  actual  malfunctions.  The  experimental 
team  felt  that  the  use  of  the  IRAN  and  NORS 
aircraft  in  no  way  compromised  the  integrity  of  the  field 
study.  Indeed,  since  a  significant  increase  was  possible 
in  the  control  which  one  could  exercise  over  the  experi¬ 
mental  situation,  it  was  felt  that  the  derived  data  had 
greater  reliability  and  validity. 

It  had  been  anticipated  originally  that  for  each  subject 
to  act  as  his  own  control,  he  would  be  assigned  the 
identical  maintenance  activity  under  ail  modes  of  infor¬ 
mation  presentation.  In  a  fair  number  of  cases,  it  was 
possible  to  accomplish  this,  but  because  of  the  exigencies 
which  prevailed  at  Charleston  --  indeed,  not  an  altogether 
surprising  situation  —  it  was  impossible  to  maintain  such 
strict  control  for  all  cases.  Later,  during  the  data 
analysis  (described  in  Section  V),  a  Mann-Whitney  U  test 
was  applied  to  data  from  the  more  controlled  cases  as  a 
check  against  the  analysis  performed  with  the  analysis 
of  variance  model. 

3.  Special  Test 

Since  the  actual  data  collection  rate  exceeded  that  which  was  originally 
anticipated  (see  Figure  4-8),  it  was  possible  to  conduct  a  special  test 
nt  the  end  of  the  basic  study.  It  was  the  purpose  of  this  study  to  con¬ 
trast  the  performance  of  unskilled  technicians  using  standard  Air  Force 
T.O.  's,  against  experienced  technicians  also  using  those  T.O.  's. 

Half  of  the  inexperienced  subjects  had  had  recent  experience  with 
similar  maintenance  actions  using  PIMO  instructions,  while  the  remain- 
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der  had  no  prior  experience  in  the  maintenance  activity  assigned.  Here 
again,  the  dependent  variables  were  Time-in-Function  (TIF)  and  errors. 
Time  data  was  collected  in  the  same  manner  as  for  the  basic  study 
described  above. 

Four  types  of  procedural  errors  were  defined  during  the  test.  These 
were: 

a.  Type  A  —  maintenance  step  was  performed  either  out  of 
sequence  or  omitted. 

b.  Type  B  --  maintenance  step  was  performed  incorrectly. 

c.  Type  C  —  indication  called  for  was  erroneously  interpreted 
and  was  not  correctly  obtained. 

d.  Type  Q  --  maintenance  action  was  terminated  due  to  lack  of 
experience  or  lack  of  information. 

This  study  was  accomplished  during  the  last  two  weeks  of  activities 
at  Charleston  Air  Force  Base,  and  only  a  sub-set  of  the  PIMO 
subjects  were  employed. 
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DATA  ANALYSIS 
A.  ANALYTICAL  MODELS 

The  statistical  techniques  for  analysis  of  the  PIMO  field  test  data  were 
selected  to  provide  precision  and  power  to  the  conclusions  and  inferences 
drawn  from  those  results.  For  the  main  study  —  comparing  the  main 
effects  of  the  PIMO  formatting  technique  --  the  analysis -of -variance 
model  was  used.  It  is  generally  accepted  that  the  factorial  analysis- 
of-variance  is  the  most  appropriate  method  for  determining  both  the 
independent  and  interactive  effects  of  two  or  mo/e  independent  vari¬ 
ables  on  a  dependent  variable.  In  the  past,  most  research  effort  was 
designed  to  study  one  independent  variable  and  its  contribution  on  the 
effect  of  the  relevant  dependent  variable.  More  recent  research  strat- 
egy.however,  calls  for  multi -variate  research  designs.  Some  author¬ 
ities  on  research  techniques  now  believe  that  muiti-variate  experi¬ 
mental  designs  are  more  valid  than  the  single  variate  type,  because  it 
more  closely  resembles  the  complex  interactions  that  one  finds  in  a 
real  world  situation  (Kerlinger,  1966). 

The  purpose  of  any  statistical  test  is  to  determine  whether  or  not  the 
null  hypothesis  should  be  rejected  on  the  basis  of  the  obtained  sample. 
The  risk  associated  with  the  decision  to  reject  the  null  hypothesis  -- 
or  failure  to  reject  the  hypothesis  --is  related  primarily  to  the  level 
of  significance  selected  as  the  criteria  for  rejection.  A3  the  level  is 
increased  (e.g. ,  from  1%  to  5%  to  10%,  etc.),  one  increases  the  risk 
of  rejecting  the  null  hypothesis  when  in  fact  there  is  no  difference  in 
the  real  situation.  Researchers  generally  use  the  1%  or  5%  level  of 
significance  in  order  to  reduce  this  risk.  *  In  a  practical  situation, 
one  is  generally  justified  in  using  even  a  higher  level  (e.g.,  8%-15%) 
depending  on  the  consequences  of  the  risk.  A  conservative  5%  level  is 

*  An  associated  risk  is  one  of  failing  to  reject  the  null  hypothesis 
when  there  is  a  ti  e  difference  in  the  real  situation. 
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used  for  the  P1MG  tost  since:  1)  any  result  must  be  examined  in  terms 
of  practical  differences,  2)  no  difference  has  as  great  a  meaning  es 
differences  —  especially  for  presentation  modes  and  skill  levels,  3) 
the  impact  of  the  test  results  are  sufficiently  large  that  the  test  must 
withstand  both  academic  and  practical  scrutiny,  and  4)  we  have  no 
basis  for  selecting  any  higher  level  except  our  own  judgment. 

B.  ANALYTICAL  PROCEDURES 

During  the  first  weeks  of  testing,  primary  analysis  began  as  soon  as 
initial  data  became  available.  Thereafter,  observed  maintenance  times 
(TIF)  were  accumulated  according  to  their  appropriate  cell  in  the  ex¬ 
perimental  design;  and,  on  a  weekly  basis,  mean  times  were  computed 
for  each  cell.'  From  this,  a  weekly  report  was  compiled  and  distributed. 
(See  Figure  5-1).  This  report  consisted  of  a  table  showing  mean  times 
and  sample  size  for  each  cell,  and  contained  graphs  which  contrasted 
the  rate  of  actual  data  collection  against  the  expected  rate  derived  under 
the  original  projections.  This  technique  provided  a  useful  management 
tool  to  assess  progress  and  to  aid  the  field  personnel. 

At  the  conclusion  of  the  field  test,  all  data  were  transferred  from  PDR 
cards  onto  IBM  keypunched  cards.  This  allowed  the  analysts  to  sort 
the  data  along  different  dimensions,  leading  to  computer  printouts 
which  presented  the  data  under  various  aspects  of  consideration,  e.g. , 
ordered  by  cells  according  to  original  matrix,  technicians,  or  mainte¬ 
nance  activities. 

The  products  of  the  analytical  procedures  discussed  below  are  present¬ 
ed  in  the  remaining  sections  of  this  report. 

1.  Basic  Study 

The  dependent  variables  taken  for  this  design  were  Time-in-Function 
(elapsed  time  to  complete  the  maintenance  activity),  and  errors.  Since 
no  errors  were  observed  for  any  maintenance  activity  accomplished 
through  the  use  of  the  PIMO  Job  Guides,  no  statistical  analysis  was 
performed. 
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A-V  I  BOOKLET 


Inspection  of  the  TIF  data  indicated  a  log  normal  distribution.  Since 
this  is  generally  the  distribution  observed  and  accepted  for  mainte¬ 
nance  performance  times,  all  scores  were  transformed  into  their  log 
equivalents. 

In  order  to  increase  the  precision  of  the  test,  it  was  decided  to  sep¬ 
arate  the  analysis  for  each  function.  This  was  reasonable,  since 
large  differences  between  the  various  functions  are  expected  simply 
due  to  the  differences  in  the  work  involved. 
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An  assumption  was  made,  supported  by  preliminary  calculations, 
that  the  observed  TIF  differences  between  maintenance  activities  were 
not  a  function  of  the  treatment  effects.  With  that  in  mind,  and  in  or¬ 
der  to  exclude  possible  obscuring  effects  due  to  the  variance  contrib¬ 
uted  by  those  different  activities,  the  factorial  design  was  partitioned 
at  the  function  level.  As  discussed  earlier,  this  led  to  two  types  of 
analysis  models  for  each  of  the  four  types  of  maintenance  activities 
(i.e. ,  Operational  Checkout,  Remove,  install  and  Adjust).  These 
were:  a)  2  x  2  x  3  models  in  which  PIMO  audio-visual  and  booklet 
modes  were  tested  with  respect  to  the  two  skill  levels  and  the  three 
function  lengths;  and  b)  three  3x3  models  where  the  two  PIMO  modes 
were  contrasted  with  Standard  Operating  Procedures.  (See  Figure 
5-2). 

The  method  of  calculation  for  the  analysis -of-variance  is  that  des¬ 
cribed  in  Lindquist  (1953). 

Specific  comparisons  made  were  as  follows: 

1.  PIMO  audio-visual  versus  PIMO  booklet  presentation 
modes  --  these  presentation  modes  were  contrasted  using 

the  performance  times  (TIF)  of  the  experienced  and  apprentice 
technicians. 

2.  Experienced  versus  Apprentice  Technicians  --  TIF  of 
the  two  groups  of  technicians  were  compared  for  both  the 
audio-visual  and  booklet  presentation  modes. 
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3.  PIMO  versus  Standard  Operating  Procedures  --  TIF  of 
experienced  technicians  using  PIMO  were  compared  to  the  same 
technicians  when  they  used  standard  procedures. 

The  model  employed  for  the  Adjust  function  was  a  2  x  2  design  -*•  A/V 
versus  Booklets;  Group  A  versus  Group  B;  and  Group  A  only  contrast¬ 
ing  A/V,  booklet,  and  SOP.  (Note:  Only  Group  A  --  (experienced)  -- 
were  permitted  to  use  SOP).  Rather  than  a  factorial  design,  a  simple 
randomized  analysis -of -variance  model  was  employed. 

As  mentioned  in  Section  IV,  the  experimental  controls  which  would 
have  allowed  each  subject  to  perform  maintenance  on  identical  tasks 
under  all  modes  of  data  presentation  were  not  possible  to  enforce. 
Consequently/  it  was  felt  that  some  doubt  might  have  been  cast  over 
the  results  gained  from  the  analysis-of-variance.  As  a  check  on 
those  results,  it  was  decided  to  perform  a  Mann-Whitney  U  test.  It 
is  understood  that  the  Mann-Whitney  U,  a  non-parametric  test  and 
one  which  requires  data  only  at  the  ordinal  level  of  measurement, 
would  not  have  the  power  of  the  analysis-of-variance.  Nonetheless, 
it  seemed  both  germane  and  important  to  test  whether  or  not  our  two 
independent  groups  had  been  drawn  from  the  same  population.  Siegel 
(1956)  states  that  the  Mann-Whitney  U  is  one  of  the  most  powerful  of 
the  non-parametrics,  "and  it  is  a  most  useful  alternative  to  the  para¬ 
metric  _t  test  when  the  researcher  wishes  to  avoid  the  _t_  test's  assump¬ 
tions,  or  when  the  measurement  in  the  research  is  weaker  than  integral 
scaling".  The  method  of  calculation  for  this  test  was  that  described 
in  Siegel. 

2.  TSA  Effectiveness  Study 

For  analysis  of  the  data  taken  from  the  TSA  Effectiveness  Study, 
student's  t_  was  selected.  It  was  used  primarily  because  there  was 
only  one  independent  variable,  and  it  is  one  of  the  most  powerful  para¬ 
metric  tests  available.  Calculations  were  performed  according  to  the 
technique  shown  in  Dixon  and  Massey  (1957). 
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3. 


User  Acceptance 


*? 

The  original  plan  was  to  use  and  other  applicable  non- parametric 
models  to  test  the  significance  of  questionnaire  responses.  However, 
inspection  of  the  data  disclosed  that  exhaustive  statistical  analysis  was 
unnecessary  since  the  responses  were  overwhelmingly  positive,  e.g. , 
90%;  100%.  Therefore,  primarily  descriptive  statistics  were  used  in 
this  report,  i.e.,  means,  percents,  etc.  with  some  analysis  using 
Fisher's  Exact  Probability  Test. 

4,  Special  Test  (PIMO  versus  Technical  Orders) 

■No  inferential  statistical  tests  were  used,  since  so  few  sample  points 
were  collected.  The  data  were  accumulated  and  rr.w  mean  performance 
time  was  calculated.  These  means  were  displayed  on  bar-graphs, 
alongside  corresponding  mean  times  from  the  basic  study. 

Error  rates  observed  during  the  special  test  were  for  a  significance 
2 

with  ^  and  Fisher's  Exact  Probability  Test. 

5.  Learning  Effects 

Two  analyses  were  performed  to  determine  the  learning  effect  present 
during  the  PIMO  field  test.  Both  of  these  are  presented  in  a  desciip- 
tive  manner,  with  no  statistical  analysis  supplied.  For  the  learning 
curves,  each  series  of  four  or  more  actions  that  involved  the  use  of 
PIMO  by  an  individual  technician,  on  a  specific  maintenance  activity, 
were  identified  and  recorded.  The  ratio  of  the  log-normal  Standard 
Operating  Procedure  time,  for  subject  activity  by  all  experienced  tech¬ 
nicians,  was  computed  and  recorded  according  to  its  sequence  in  the 
performance  series,  i.e.,  first  time  performed,  second  time  perform¬ 
ed,  and  so  forth.  The  ratios  were  averaged  for  experienced  and  in¬ 
experienced  technicians  by  the  repetitive  action  number  and  plots  were 
drawn  for  each  experience  level  respectively. 
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In  addition  to  the  learning  curves  taken  from  the  performance  times, 
a  time  change  matrix  was  generated  in  the  form  of  a  three-by-three 
contingency  table.  Plotted  in  this  matrix  were  the  performance  times 
by  presentation  mode  for  each  occurrence  of  a  particular  maintenance 
activity  as  repeated  by  an  individual  technician.  For  each  successive 
pair  of  actions,  i.e.,  first  and  second,  second  and  third,  etc.,  the 
difference  in  the  log  performance  times  was  recorded  on  the  matrix. 
Following  this,  the  average  for  each  cell  was  calculated  and  the  per¬ 
centage  of  time  change  computed.  It  was  felt  that  these  time  change 
matrices  would  offer  supporting  evidence  for  the  occurrence  of  learning 
as  demonstrated  by  the  learning  curves. 

6.  Systems  Effectiveness/AMES  Model 

A  determination  of  change  in  over-all  systems  effectiveness  as  a 
function  of  PIMO  application  was  estimated  through  the  use  of  the  AMES 
Model.  AMES,  an  acronym  for  Aircraft  Maintenance  Effectiveness 
Simulation,  was  developed  early  in  the  program  by  Serendipity,  Inc. 

It  is  a  digital  simulation  model  programmed  for  the  IBM  7094  and 
represents  a  technique  of  simulating  a  real  system  by  means  of  a  dy¬ 
namic  mathematical  representation  of  that  system.  The  AMES  simu¬ 
lation  model  is  discussed  more  fully  in  Section  VII  of  this  volume.  In 
general,  however,  the  AMES  Model  accepts  data  which  represent  the 
aircraft  in  terms  of  its  major  subsystems  and  their  principal  components. 
For  example,  failure  rates,  repair  policy,  spares  level,  and  data  re¬ 
flecting  scheduled  and  unscheduled  maintenance  for  systems  and  sub 
systems  are  furnished  to  the  simulation  program.  Furthermore,  datu 
are  provided  which  specify  the  required  types  and  numbers  of  personnel 
necessary  to  perform  maintenance.  Task  performance  times,  and 
personnel  reliability  are  also  included  in  the  data  bank.  The  simulation 
model  is  driven  by  impressing  mission  demands.  Results  obtained 
from  the  various  runs  of  this  model  are  reported  in  Section  VI  of  this 
volume. 
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SECTION  VI 


l 


TEST  RESULTS 


A.  INTRODUCTION 


Stated  broadly,  the  main  purpose  of  the  PIMO  field  test  was  to  deter¬ 
mine  whether  or  not  the  newly  formatted  maintenance  instructions 
could  be  instrumental  in  aircraft  maintenance  improvement.  Since 
the  major  areas  of  consideration  were:  better  utilization  of  resources 
(i.e.,  manpower),  and  reduction  of  costly  and  often  dangerous  mainte¬ 
nance  performance  errors,  it  is  with  emphasis  on  those  points  that 
the  results  of  the  data  analysis  will  be  presented. 

B.  PERFORMANCE  DATA 

Since  no  error  was  noted  in  over  1,  000  observations  of  performance 
with  PIMO,  no  comparison  of  error  rate  was  made.  Thus,  the  dis¬ 
cussion  of  the  results  is  limited  to  performance  time.  Previous  anal¬ 
ysis  of  maintenance  records  showed  that  the  "normal"  maintenance 
results  in  a  fairly  high  error  rate.  Thus,  it  can  be  stated  with  a  fairly 
high  degree  of  confidence  that  PIMO  resulted  in  a  100%  reduction  of 
error  rate  for  the  test.  Any  improvement  in  performance  time  dis¬ 
cussed  in  this  subsection  should  be  viewed  as  an  additional  improve¬ 
ment  beyond  the  100%  error  reduction, 

1.  PIMO  versus  Standard  Operating  Procedures 

The  data  foi  performance  times  were  converted  to  log-normal  equiv¬ 
alents,  and  the  mean  time  for  each  condition  was  calculated.  Those 
mean  times  are  presented  in  bar  graph  form  in  Figure  6-1  and  6-2, 

It  should  be  noted  that  only  experienced  technicians,  i.e.,  5-level, 
performed  maintenance  using  Standard  Operating  Procedures.  This 
was  because  it  seemed  to  make  no  sense  to  allow  an  apprentice  tech¬ 
nician  to  perform  maintenance  using  procedures  for  which  he  had  no 
experience  or  training.  Consequently,  for  comparison  of  Standard 
Operating  Procedures  only  the  5-level,  certified  technicians  were 
employed. 


EXPERIENCED  TECHNICIANS 
I.  OPERATIONAL  CHECKOUTS 


A-V 


Figure  6-1  PIMO  Audio-Visual  vs.  PIMO  Bookie?*:  Presentation  Modes  vs. 

Standard  Operating  Procedure 
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The  data  show,  as  illustrated  in  the  figures,  that  the  maintenance 
times  were  shorter  for  Standard  Operating  Procedures  than  for  PIMO 
under  either  A-V  or  booklet  presentation  mode.  This  is  not  altogether 
unexpected,  since  the  mean  times  do  not  reflect  the  level  of  perfor¬ 
mance  eventually  exhibited  by  the  technicians  subsequent  to  their 
learning.  As  will  be  seen  later,  it  is  possible  for  all  technicians  to 
approach  the  Standard  Operating  Procedure  maintenance  times  as  an 
asymptote.  Thus,  the  average  time  with  PIMO  is  a  function  of  the 
number  of  samples  obtained  for  a  given  activity.  If  we  had  conducted 
the  test  for  a  longer  period  of  time  and  the  learning  curve  observed 
held  true,  the  PIMO  time  would  probably  be  the  same  as  tne  SOP  time. 

The  mean  times  shown,  do  reflect  that  on  the  average,  SOP  require 
35%  less  time  than  the  PIMO  audio-visual  mode  and  18%  less  time 
than  the  PIMO  booklet  mode  of  information  presentation. 

(The  data  collected  for  the  effectiveness  of  the  troubleshooting  aids 
on  the  other  hand,  reflect  that  the  use  of  these  aids  resulted  in  an  11% 
decrease  in  troubleshooting  time  (see  Figure  6-3)). 

There  are  some  significant  comparisons  to  which  the  reader's  attention 
should  be  directed.  Short  Operational  Checkouts,  for  example,  indicate 
that  there' is  a  .  7  minute  real  difference  between  booklets  and  the  SOP, 
giving  a  difference  of  7.  6%;  and  a  real  difference  of  two  minutes  between 
A-V  and  SOP  for  a  16,  8%  difference.  In  a  practical  sense,  th  s  would 
indicate  that  the  performance  times  with  the  PIMO  presentation  mode 
do  indeed  approximate  those  of  the  SOP.  This  is  quite  dramatic  when 
one  considers  that  the  performance  times  described  are  mean  times, 
as  mentioned  above,  and  do  not  reflect  the  steady  reduction  as  the  sub¬ 
jects  learned.  Other  comparisons  are  equally  dramatic.  For  example, 
medium  Remove  activities  indicate  real  differences  of  approximately 
six  minutes  between  audio-visual  and  SOP,  and  2.4  minutes  between 
booklet  and  SOP.  For  short  Remove  activities,  the  time  differences 
are  even  less  --  1.  9  for  A-V  versus  SOP,  and  .  7  minutes  for  booklet 
versus  SOP.  The  other  comparisons  are  also  indicative  of  the  close - 


Vol.2 


6-4 


ness  of  the  performance  times  between  presentation  modes  and  the 
standard  practices. 

On  the  other  hand,  there  are  other  instances  reflected  in  Table  6-1 
which  indicate  that  for  some  activities  there  is  divergence  in  perfor¬ 
mance  times.  A  good  example  is  that  of  long  Operational  Checkouts. 
Here,  the  time  differences  between  A-V  and  SOP  are  44  minutes,  for 
an  increase  of  80.6%;  and  a  time  difference  of  42. 1  minutes  for  an 
increase  of  79.  9%  for  booklets  versus  SOP.  There  is  reason  to  believe 
that  this  difference  is  due  to  shortcuts  taken  by  the  experienced  tech¬ 
nicians  performing  without  data.  These  shortcuts  may  or  may  not  pro¬ 
vide  an  adequate  check  of  the  system  of  concern. 

2.  Comparison  of  Audio-Visual  versus  Booklet  Presentation  Mode 

Figures  6-4  and  6-5  show  the  comparisons  for  audio-visual  versus 
booklet  presentation  techniques,  within  the  apprentice  technician  group. 
These  bar  charts  were  generated  in  much  the  same  way  as  those  des¬ 
cribed  above.  That  is,  log  normal  performance  times  in  minutes  were 
accumulated,  their  means  calculated  and  the  bar  charts  drawn.  From 
observation,  one  can  quickly  see  that  in  most  cases  there  is  relatively 
little  real  difference  in  the  performance  times  for  any  activity,  or 
activity  duration,  for  audio-visual  or  booklet  modes.  In  two  instances, 
however,  the  times  are,  in  terms  of  their  percentages,  somewhat 
large.  Those  two  instances  are  for  medium  Install  activities  --  where 
the  mean  time  for  A-V  was  6. 6  minutes  and  the  mean  time  for  booklets 
was  4.4  minutes  --  and  medium  Remove  activities,  where  the  mean 
time  for  A-V  was  8.  7  minutes  and  the  mean  time  for  booklets  was  6. 2 
minutes. 

Once  again,  one  should  be  cautious  in  interpreting  these  data  as  repre¬ 
sentative  of  performance  times  which  one  might  experience  from  the 
technicians  in  tho  field.  Since  they  are  means,  they  include  the  times 
during  which  learning  occurred,  and  they  are  thus  weighted  in  a  direction 
which  produces  longer  performance  times. 
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OPERATIONAL  CHECKOUT 


PRESENTATION  MODE 


COMPARISON 


BOOKLET  VS.  SOP 


PERCENT 

DIFFER. 


PERCENT  between^  PERCENT 
DIFFER.  MEANS  DIFFER. 
IN  MINUTES 


-32.4 


-50.0 


-24.0 


BOOKLET  VS.  SOP 


PERCENT 

DIFFER. 

DIFFERENCE 

BETWEEN 

MEANS 

IN  MINUTES 

PERCENT 

DIFFER. 

+27.  6 

+  1.3 

+27.6 

-62.9 

-2.  1 

-47.7 

-18.  2 

+0.2 

+8.0 

COMPARISON 


BOOKLET  VS.  SOP 


PERCENT 

DIFFER. 

DIFFERENCE 

BETWEEN 

MEANS 

IN  MINUTES 

PERCENT 

DIFFER. 

-  10.5 

-14.6 

-10.  7 

Table  6-1  Within  Group  Comparisons,  Group  A  (Experienced  Technicians) 
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APPRENTICE  TECHNICIANS 
3.  INSTALL  ACTIVITIES 


4.  ADJUST  ACTIVITIES 


Figure  6-5  PIMO  Audio-Visual  vs.  PIMO  Booklet 


Tne  graphs  shown  indicate  that  in  four  of  the  ten  cases,  the  use  of 
the  audio-visual  mode  of  presentation  resulted  in  shorter  mean  times 
than  the  use  of  the  booklet.  But  it  should  be  recognized  that  the  over¬ 
all  difference  of  3%,  while  still  small,  is  in  favor  of  the  booklets. 

Table  6-2  reflects  the  log  normal  means  and  the  mean  differences  be¬ 
tween  the  two  modes  of  presentation  for  the  apprentice  technicians. 

One  can  see  that  the  practical  difference  of  performance  times  is  very 
low  --  in  most  cases,  below  four  mirutes.  Only  in  two  cases  does  the 
difference  of  performance  time  exceed  four  minutes  --  and  that  is  for 
Adjust  activities  and  long  Operational  Checkouts.  Since  there  is  a 
definite  trend  towards  improved  performance  with  use  of  the  booklets, 
coupled  with  the  fact  that  audio-visual  devices  require  more  production 
resources  (time,  hardware,  etc. )  this  would  lead  one  to  accept  booklets 
as  the  better  presentation  mode  --  for  apprentice  technicians,  at  least. 

3.  Experienced  versus  Apprentice  Technicians 

The  test  results  which  contrasted  the  skill  levels  indicate  that  in  general, 
experienced  technicians  take  less  time  to  perform  a  given  maintenance 
task  than  apprentices.  And,  while  this  holds  true  for  both  PIMO  modes, 
it  is  also  obvious  from  the  graphs  (see  Figures  6-6,  6-7,  6-8,  6-9)  that 
the  differences  between  the  performance  times  of  the  two  groups  were 
not  as  marked  for  the  audio-visual  as  they  were  for  the  booklet  mode. 

The  average  time  increase  equalled  8%  with  the  use  of  the  audio-visual 
mode,  and  33%  with  the  booklet  mode.  This  has  significant  impact  on 
over-all  system  performance,  since  it  suggests  that  it  is  now  possible 
to  employ  inexperienced  technicians  to  perform  specialized  maintenance. 
This  is  true,  almost  irrespective  of  the  presentation  mode,  but  seems 
to  hold  more  fox’  booklets  than  for  audio-visual  presentation.  It  is  in¬ 
teresting  to  note  that  in  some  cases  the  differences  in  performance 
times  were  almost  negligible;  in  specific:  short  and  medium  Operational 
Checkouts  using  booklets,  medium  Install  activity  using  booklets  or 
audio-visual,  and  medium  Remove  activities  using  audio-visual.  (See 
Table  6-3).  Here  again,  one  must  be  cautioned  when  in  drawing  con- 
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Visual  &  Using  PIMO  Booklet 


V 

X 

T 


OPERATIONAL  CHECKOUT 


PRESENTATION  MODE 


LOG  NORMAL 
MEANS  IN 
MINUTES 


COMPARISON 
A-V  VS.  BOOKLETS 


DIFFERENCE  De»rcMT 

BOOKLET  BETWEEN  MEANS 

IN  MINUTES  DIFFERENCE 


LONG 

69.8 

62.  8 

-7.  J 

BTJ  1 

MEDIUM 

24.3 

27.4 

+3.1 

+14.8 

SHORT 

13.4 

9.8 

-3.6 

-26.9 

REMOVE  ACTIVITIES 


PRESENTATION  MODE 


LOG  NORMAL 
MEANS  IN 
MINUTES 


COMPARISON 
A-V  VS.  BOOKLETS 


DIFFERENCE  owreKir 
BOOKLET  BETWEEN  MEANS  'JKIiSL 


IN  MINUTES 


DIFFERENCE 


LONG 

17.9 

15.8 

-2.  1 

-11.7 

MEDIUM 

8.7 

6.2 

-2.5 

-28.7 

SHORT 

4.8 

5.6 

+1.2 

+25.0 

INSTALL  ACTIVITIES 


PRESENTATION  MODE 


LOG  NORMAL 
MEANS  IN 
MINUTES 


COMPARISON 
A-V  VS.  BOOKLETS 


DIFFERENCE 
BETWEEN  MEANS 
IN  MINUTES 

PERCENT 

DIFFERENCE 

+.4 

+5.  j 

-2.2 

-33.  3 

+1. 1 

+23.9 

ADJUST  ACTIVITIES 


PRESENTATION  MODE 


LOG  NORMAL 
MEANS  IN 
MINUTES 


COMPARISON 
A-V  VS.  BOOKLETS 


137.9 


DIFFERENCE  D„rCK1T 

BOOKLET  BETWEEN  MEANS  mptwIirF 
IN  MINUTES  DIFFEREN*  ... 


133.3 


Table  6-2  Within  Group  Comparisons  (Group  B,  Inexperienced  Technicians) 
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OPERATIONAL  CHECKOUT 


AUDIO-VISUAL  DEVICE 

BOOKLET 

LOG  NORMAL 
MEANS  IN 
MINUTES 

COMPARISON 
EXPERIENCED  VS. 
INEXPERIENCED 

LOG  NORMAL 
MEANS  IN 
MINUTES 

COMPARISON 
EXPERIENCED  VS. 
INEXPERIENCED 

a 

-  JS 

DIFFERENCE 
RETWEEN  MEANS 
IN  MINUTES 

PERCENT 

DIFFER. 

EXP'D. 

TECH. 

BBS 

PERCENT 

DIFFER. 

LONG 

54,6 

69.9 

+  15.  3 

+  18.6 

52.7 

62.8 

+  10. 1 

+19.2 

MEDIUM 

23.0 

24.3 

+  1.3 

KB 

28.5 

27.4 

-  1.1 

-  3.9 

SHORT 

11.9 

13.4 

+  1.5 

+12.6 

9.2 

9.8 

+  •  6 

EQ3 

REMOVE  ACTIVITIES 


AUDIO-VISUAL  DEVICE 

BOOKLET  | 

LOG  NORMAL 
MEANS  IN 
MINUTES 

COMPARISON 
EXPERIENCED  VS. 
INEXPERIENCED 

LOG  NORMAL 
MEANS  IN 
MINUTES 

COMPARISON 
EXPERIENCED  VS. 
INEXPERIENCED 

H 

1 

DIFFERENCE 
BETWEEN  MEANS 
IN  MINUTES 

PI  CENT 
Dir'FER. 

EXP'D. 

TECH. 

INEXP'D. 

TECH. 

PERCENT 

DIFFER. 

LONG 

15.0 

17.9 

+2.9 

+  19.3 

11.4 

15.  8 

+4.4 

+38.6 

MEDIUM 

8.3 

8.7 

+0.  4 

+4.  8 

4.8 

6.2 

+  1.4 

+29.  1 

SHORT 

4.1 

4.8 

+0.  7 

+  17.0 

2.9 

5.6 

+3.  7 

+127. 1 

INSTALL  ACTIVITIES 


AUUIO-V'SUAL  DEVICE 

BOOKLET 

LOG  NORMAL 
MEANS  IN 
MINUTES 

COMPARISON 
EXPERIENCED  VS. 
INEXPERIENCED 

LOG  NORMAL 
MEANS  IN 
MINUTES 

COMPARISON 
EXPERIENCED  VS. 
INEXPERIENCED 

a 

BBS 

mm 

DIFFERENCE 
BETWEEN  MEANS 
IN  MINUTES 

PERCENT 

DIFFER, 

EXP'D. 

TECH. 

INEXP'D. 

TECH. 

DIFFERENCE 
BETWEEN  MEANS 
IN  MINUTES 

PERCENT 

DIFFER. 

15  :  | 

KSB 

7.2 

+2.5 

+53.  2 

wsm 

7.6 

+2.9 

+61.9 

MEDIUM 

8.2 

6.6 

+0.  4 

+6.4 

EH 

KSB 

0 

0 

SHORT 

3.3 

4.6 

+1.3 

+39.3 

2.5 

+3.2 

+128.0 

ADJUST  ACTIVITIES 


AUDIO-VISUAL  DEVICE 

BOOKLET 

LOG  NORMAL 
MEANS  IN 
MINUTES 

COMPARISON 
EXPERIENCED  VS. 
INEXPERIENCED 

LOG  NORMAL 
MEANS  IN 
MINUTES 

COMPARISON 
EXPERIENCED  VS. 
INEXPERIENCED 

m 

I"  ' 

E  ■ 

ggH 

PERCENT 

DIFFER. 

EXP'D. 

TECH. 

INEXP'D. 

TECH. 

DIFFERENCE 
BETWEEN  MEANS 
IN  MINUTE? 

PERCENT 

DIFFER. 

136.3 

137.9 

+1.6 

+1.2 

136.6 

133.3 

•3.3 

-2.4 

Table  6-3  aeiwcca  uiuup  comparisons 


elusions  from  these  data  since  they  do  represent  mean  times  uncor¬ 
rected  for  learning.  One  could  safely  assume  that  where  little  or  no 
practical  differences  in  performance  times  exist,  they  would  remain 
so,  subsequent  to  the  learning  phase.  On  the  other  hand,  demonstrated 
elsewhere  is  the  fact  that  learning  does  occur.  It  would  be  reasonable 
to  assume  therefore  that  one  would  see  a  reduction  in  apprentice  per¬ 
formance  times,  to  the  point  where  they  begin  to  approximate  the  times 
of  experienced  technicians. 

If  one  were  to  assume  that  these  average  times  were  to  remain  constant, 
this  would  not  negate  the  potential  effect  that  is  demonstrated  here. 

Not  the  least  aspect  of  this  effect  is  that  maintenance  is  performed  by 
apprentice  technicians,  not  so  much  that  it  is  performed  in  some  time 
which  is  greater  than  that  for  experienced  technicians.  In  a  practical 
sense,  when  one  sees  rather  large  differences  between  performance 
times  of  the  two  groups  —  for  example,  a  long  Install  activity  on  audio¬ 
visual  where  the  percentage  differences  are  as  large  as  80%  —  this  is 
not  an  unacceptable  increase  in  performance  time.  One  must  consider 
that  it  is  far  superior  to  having  no  maintenance  at  all. 

Certainly  this  is  not  to  suggest  that  all  maintenance  be  assigned  to  in¬ 
experienced  technicians.  But  one  can  see  clearly  that  an  appropriate 
strategy  for  those  situations  wherein  the  experienced  technician  cadre 
are  saturated,  would  be  to  assign  some  cf  the  less  critical  maintenance 
activities  to  the  inexperienced  technician. 

STATISTICAL  ANALYSIS 

1.  Basic  Study  Calculations 

Subsequent  to  the  plotting  of  data  for  descriptive  purposes  the  analysis- 
of -variance  tests  were  calculated.  Summaries  of  these  tests  for  each 
of  the  four  types  of  maintenance  activities  appear  in  Tables  6-4,  6-5, 

6-6  and  6-7.  As  mentioned  earlier,  the  significance  stipulated  was 
the  .  05  level  and  for  a  one  tailed  test. 


SOURCE  OF  VARIATION 

SS 

df 

MS 

F  RATIO 

A  {PRESENT.  MODE) 

.0464 

1 

.0464 

.6946 

B  (SKILL  LEVEL) 

.1287 

1 

.  1287 

1.9266 

C  (FUNCTION  LENGTH) 

22. 8893 

2 

11.4446 

171.3263* 

AB 

1 

.0190 

.2  844 

AC 

.4115 

2 

.2058 

3.0808 

BC 

.0823 

2 

.  0412 

.6168 

ABC 

2 

3.  1629 

46.8997* 

WITHIN  CELL 

16.9622 

254 

.0668 

*  Significait  at  the  .05  level  for  a  one-tailed  test  of  significance. 


SOURCE  OF  VARIATION 

SS 

df 

MS 

F  RATIO 

A  (PRESENT.  MODE) 

2 

3.2178 

38.6291* 

C  (FUNCTION  LENGTH) 

6.4246 

2 

3.2123 

38.5630* 

AC 

3.2141 

4 

.  8035 

9.  6459* 

WITHIN  CELL 

13. 4998 

162 

.0835 

*  Significant  at  the  .05  level  for  a  one-tailed  test  of  significance. 


F  RATIOS 

Fo95(1,00)  =  3.84 
F.95{^00)  =  2.99 
F.95(2/162)=3  05 
F.9j<4,162)=2.43 


Table  6-4  Summary  of  Analysis  of  Variance  Calculations  for  Operational 

Checkout 
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SOURCE  OF  VARIATION 

SS 

df 

MS 

F  RATIO 

A  (PRESENT.  MODE) 

HSH 

1 

.2653 

2.7751 

B  (SKILL  LEVEL) 

1.9698 

1 

1. 9698 

20.6046* 

C  (FUNCTION  LENGTH) 

2.0787 

2 

1.0398 

10.8766* 

AB 

.0965 

1 

.0965 

1.0094 

AC 

.4851 

2 

.2426 

2.5377 

BC 

.8442 

2 

.4221 

4.4153* 

ABC 

.  2278 

2 

.1139 

1.1914 

WITHIN  CELL 

33.5432 

351 

.0956 

*  Significant  at  the  .05  level  for  a  one-tailed  test. 


SOURCE  OF  VARIATION 

SS 

df 

MS 

F  RATIO 

A  (PRESENT.  MODE) 

.  9143 

2 

.4572 

4.  1376* 

C  (FUNCTION  LENGTH) 

3.2282 

2 

1.6141 

14.  6072* 

AC 

2.5222 

4 

.6306 

5.7068* 

WITHIN  CELL 

_ _ _ _ _ 

32.5960 

295 

_ M,J 

.1105 

F  RATIOS 

F. 95(1,00)  =  3.84 
F.  95(2, 00)  =  2.99 
F.  95(4, 00)  =  2,37 


Table  6-5  Summary  of  Analysis  of  Variance  Calculations  for  Install 
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SOURCE  OF  VARIATION 

SS 

df 

MS 

F  RATIO 

A  (PRESENT.  MODE) 

.9139 

1 

.  9139 

11.2688* 

B  (SKILL  LEVEL) 

1. 1206 

1 

1. 1206 

13.8175* 

C  (FUNCTION  LENGTH) 

16.3118 

2 

8. 1559 

100.5660* 

AB 

.2910 

1 

.2910 

3.5882 

AC 

.3291 

2 

.  1646 

2.0296 

BC 

.1822 

2 

.0911 

1.1233 

ABC 

.0843 

2 

.0422 

0.5203 

WITHIN  CELL 

28.7935 

355 

.0811 

*  Significant  at  the  .  05  level  for  a  one-tailed  test  of  significance 


SOURCE  OF  VARIATION 

SS 

df 

MS 

F  RATIO 

A  (PRESENT.  MODE) 

6.4967 

2 

3.2484 

32. 1943* 

C  (FUNCTION  LENGTH) 

15.1222 

2 

7.5611 

74. 9366* 

AC 

.  6594 

4 

.  1649 

1.6343 

WITHIN  CELL 

30. 7856 

305 

.  1009 

*  Significant  at  the  .  05  level  for  a  one-tailed  test  of  significance. 


F  RATIOS 

F.95(JP0)  3.84 
F.  95(2,0 0)=2.99 
F.95(4,00)=2.37 


Table  6-6  Summary  of  Analysis  of  Variance  Calculations  for  Remove 
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Table  6-4  summarizes  the  calculations  of  the  analysis- of- variance 
for  the  Operational  Checkout  maintenance  function.  As  shown  at  the 
bottom,  an  F-ratio  of  3.  84  is  required  for  significance  with  one  and 
254  degrees  of  freedom  (as  shown  in  some  cases);  and  an  F-ratio  of 
2.  99  is  required  with  two  and  254  degrees  of  freedom.  Notice  that 
only  in  three  instances  do  we  have  an  F-ratio  significant  at  the  .  05 
level  of  higher.  In  one  case  we  have  an  F-ratio  of  171,  which  shows 
an  extreme  degree  of  statistical  significance,  but  with  no  practical 
significance  since  it  is  associated  with  function  length  (i.e.,  short, 
medium  and  long).  It  is  fully  expected  that  the  differences  in  function 
length,  as  an  inherent  quality  of  the  maintenance  activity,  would  in¬ 
deed  show  significance.  However,  in  the  other  two  cases  --  presenta¬ 
tion  mode  by  function  length  interaction,  and  presentation  mode  by  skill 
level  by  function  length  interaction  --  there  is  practical  significance. 

What  may  not  appear  quite  so  obvious  is  the  importance  of  the  fact  that 
there  is  no  significance  between  skill  levels.  At  first  glance,  since 
one  is  normally  used  to  looking  for  significance,  this  would  appear  to 
be  a  negative  finding.  But,  closer  inspection  will  reveal  that  one  should 
normally  expect  to  find  a  significant  difference  in  performance  times 
between  these  two  groups,  since  there  is  a  real  difference  in  their 
experience  level.  This  further  reinforces  the  idea  that  unskilled  mainte¬ 
nance  technicians  may  be  assigned  to  those  maintenanc  tasks  hereto¬ 
fore  reserved  for  the  qualified  technician  only.  Furthermore,  since 
there  appears  to  be  no  significance  between  the  two  presentation  modes, 
this  would  support  the  conclusion  that  the  visual-only,  or  booklet  mode 
of  presentation,  would  be  the  more  cost  effective. 

The  lower  half  of  Table  6-4  indicates  the  within  group  comparison  of 
Group  A  only  using  the  audio-visual,  booklet,  and  standard  operating 
procedures  for  long,  medium,  and  short  operational  checkouts.  No 
comparison  is  made  between  these  three  by  three  independent  variables 
and  the  unskilled  group,  since  unskilled  maintenance  technicians  were 
not  allowed  to  perform  maintenance  using  SOP.  Notice  also  that  aU 
elements  show  significance.  Once  again,  the  reader  is  cautioned  to 
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disregard  the  significance  shown  for  function  length,  since  this  was  an 
inherent  characteristic  of  the  test  design. 

There  is,  as  mentioned  before,  significance  within  the  main  effect  of 
presentation  mode.  This  significance  is  created  primarily  from  dif¬ 
ferences  in  performance  times  between  PIMO  and  SOP.  As  described 
earlier,  and  as  reflected  in  the  bar  charts,  the  mean  time  to  perform 
maintenance  under  SOP  was  shorter  than  that  for  maintenance  perform¬ 
ed  under  PIMO.  Once  again  this  was  due  to  an  artifact  in  the  data 
analysis,  since  all  data  were  accumulated  to  produce  irieiai  times,  and 
mean  times  do  not  reflect  the  performance  of  either  inexperienced  or 
experienced  technicians  once  they  have  had  an  opportunity  to  acquire 
learning. 

Figure  6-10  reflects  the  presentation  mode  by  function  length  inter¬ 
action  for  Operational  Checkout  activities.  The  large  variance  exists 
essentially  for  long  and  medium  activities,  when  audio-visual  mode  is 
contrasted  with  booklet.  While  statistical  significance  does  occur  as 
a  consequence  of  the  differences  oetween  these  means,  it  can  be  seen 
that  the  practical  difference  is  not  enough  to  warrant  the  absolute  re¬ 
jection  of  one  presentation  mode  over  the  other  --  at  least  not  based 
on  mean  performance  times  alone.  As  mentioned  many  times  earlier, 
however,  when  the  cost  factors  are  introduced,  it  becomes  obvious 
that  the  booklet  mode  is  the  more  effective. 

Table  6-5  is  a  summary  of  the  analysis-of -variance  calculations  for  the 
Install  maintenance  function.  The  same  level  of  significance  for  one 
tailed  test  was  imposed,  and  the  required  F-ratios  are  shown  at  the 
bottom  of  the  table.  For  the  2x2x3  analysis,  there  were  only  three 
elements  of  significance:  skill-level  and  function-length  main  effects, 
and  the  skill-level  by  function-level  interaction.  Function  length  sig¬ 
nificance  again  is  discounted.  The  significance  of  the  skill-level  main 
effect  is  important.  Contrary  to  the  skill-level  main  effect  for  Opera¬ 
tional  Checkout,  the  significance  here  indicates  that  there  is  a  difference 
in  performance  time  by  the  two  groups  --  experienced  and  inexperienced 


technicians.  Figure  6-11  plots  the  skill-level  by  function-length  inter¬ 
action.  It  is  obvious  that  the  long  and  short  Install  activities  contribute 
the  major  source  of  variance,  with  the  inexperienced  technician  taking 
considerably  longer  on  both  of  those  activities.  A  satisfactory  expla¬ 
nation  to  account  for  these  differences  would  be  difficult  to  make,  based 
on  the  data  available.  Nonetheless,  when  one  takes  a  look  at  the  real 
times  involved  --  approximately  a  three  minute  difference  on  long  In¬ 
stalls  and  approximately  2-1/2  minutes  on  the  short  Installs  --  one 
can  see  that  there  is  very  little  practical  difference  indeed.  The  small 
penalty  that  one  might  be  called  upon  to  pay  for  the  use  of  inexperienced 
technicians  would  be  paid  back  by  an  over-all  improvement  in  systems 
effectiveness  as  a  consequence  of  more  aircraft  available  to  accept 
mission  assignments. 

The  3x3  analysis  --  Group  A  only,  contrasting  audio-visual  with  book  ¬ 
let  and  SOP  for  long,  medium,  and  short  activities  —  indicates  sig¬ 
nificance  at  both  main  and  interaction  effects.  A  review  of  Figure  6-2 
points  up  an  interesting  anomaly.  For  short  and  medium  activities, 
SOP  times  are  equal  to,  or  lower  than,  the  PIMO  performance  times. 
However,  with  the  long  Install  activity,  SOP  times  become  greater  than 
either  of  the  two  PIMO  presentation  modes.  This  is  particularly  in¬ 
teresting  in  light  of  the  fact  that  here  again  the  over-all  means  include 
the  learning  times.  The  difference  between  SOP  and  A-V  times  for 
medium  Install  acti  'ities  is  almost  a  factor  of  three.  This  would  seem 
at  first  to  rule  out  the  use  of  PIMO  in  the  A-V  mode,  but  we  are  des¬ 
cribing  a  real  difference  of  only  4  minutes.  It  would  be  safe  to  assume 
that  the  GOP  average  of  2.3  minutes  represents  a  time  fairly  close  to 
the  physiological  limit  of  performance  on  these  activities.  It  would 
also  seem  reasonable  to  expect  that  the  performance  time  for  this 
activity  on  the  A-V  or  booklet  presentation  mode  would  begin  to  drift 
toward  that  level,  subsequent  to  initial  learning. 

The  summary  for  the  analysis -of -variance  for  the  Remove  maintenance 
activity  is  shown  in  Table  6-6.  Once  again,  the  same  significant  level 


was  selected  for.  one  tailed  test;  and  the  f-ratios  required  for  those 
significances  are  shown  at  the  bottom  of  the  table.  The  elements  of 
significance  are:  the  presentation  mode,  skill  level  and  function  length 
main  effects.  There  is  no  significance  for  the  interaction  effects. 
Function  length,  again,  will  not  be  discussed  for  the  reasons  mentioned 
above.  Review  of  Figure  6-1  shows  that  there  is  a  large  contribution 
made  by  the  differences  in  performance  time  between  A-V,  booklet  and 
SOP  for  both  inexperienced  and  experienced  technicians.  It  should  be 
noted  that  the  differences  in  performance  times  for  each  of  these  modes 
is  greater  in  degree  for  the  experienced  and  inexperienced  technicians. 
Furthermore,  the  direction  of  differences  is  the  same  in  all  cases, 
except  that  of  short  Remove  activities. 

For  apprentice  technicians,  the  difference  between  A-V  and  booklet  is 
reversed  from  that  of  the  experienced  technicians.  In  all  other  cases, 
A-V  presentation  mode  requires  the  longest  performance  times.  The 
greatest  difference  in  significant  performance  times  is  for  long  Remove 
activities  by  experienced  technicians  --  where  the  time  goes  from  7.3 
to  15  minutes. 

The  differences  in  skill  level  performance  are  not  as  drastic  as  one 
might  assume  from  the  statistical  signiiicance  which  was  achieved.  In 
most  cases,  less  than  2  or  3  minutes  represent  the  difference  between 
experienced  and  inexperienced  personnel  performance.  The  largest 
difference  occurs  for  long  Remove  activities  using  the  PIMO  booklets, 
where  the  experienced  personnel  perform  in  approximately  11  minutes, 
as  opposed  to  the  apprentice  personnel  in  approximately  16  minutes. 

The  practical  difference  between  these  two  times  is  negligible  when 
considered  in  light  of  the  over-all  system  effectiveness. 

The  3x3  table  indicates  that  there  is  a  significance  in  the  main  effect 
of  presentation  mode.  This  is  primarily  due  to  the  differences  noted 
between  the  technicians  using  SOP  and  either  one  of  the  PIMC  pre¬ 
sentation  modes. 


The  analysis -of- variance  calculations  performed  on  the  Adjust  mainte¬ 
nance  function  is  shown  in  Table  6-7.  Only  a  2  x  2  analysis  was  per¬ 
formed  rather  than  the  2x2x3,  since  it  was  not  possible  to  collect  a 
sufficient  number  of  sample  points  to  adequately  perform  the  larger 
analysis.  There  are  no  statistical  significances  shown  here,  but  again 
this  in  itself  has  some  significance,  since  it  indicates  that  inexperienced 
personnel  can  perform  Adjust  maintenance  functions  as  well  as  the 
skilled.  The  3x1  table  shows  significance  for  the  presentation  mode 
main  effect,  primarily  as  a  consequence  of  the  variance  contributed 
by  the  differences  between  PIMO  and  SOP. 

2,  Basic  Study  Summary 

In  relating  the  results  of  the  analysis -of-variance  tests  to  the  data 
configurations  just  described,  the  following  summary  may  be  given. 

a.  Presentation  Modes 

While  the  booklet  performance  time  averages  observed  during 
this  test  were,  on  the  whcle,  shorter  than  for  the  audio-visual 
mode,  the  difference  was  statistically  significant  only  for  the 
Remove  activities.  This  would  indicate  that  the  technicians 
reflected  no  preferences  in  the  presentation  modes,  as  a  func¬ 
tion  of  their  performance  times.  Therefore,  it  would  be  safe 
to  assume  that  one  could  propose  as  an  operational  system,  the 
least  expensive  of  the  two  modes.  The  booklet  mode  --  visual 
only  --  has  been  demonstrated  to  be  somewhat  less  expensive 
in  all  aspects  than  audio-visual  presentation.  Consequently, 
it  is  that  technique  which  is  recommended  in  Volume  III  of 
this  report.  Operational  System  Design. 

b.  Skill  Level 

The  results  of  the  analysis  indicate  that  apprentice  technicians 
do,  in  most  cases,  require  more  time  to  perform  a  given  mainte¬ 
nance  function  than  the  certified  specialists.  Those  differences. 
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however,  while  statistically  significant  only  in  the  case  of  Remove 
sind  Install  activities  do  represent  rather  small  practical  differ¬ 
ences.  The  f-ratios  obtained  from  the  analysis-of-variance  sug¬ 
gest  that  no  significance  may  be  attached  to  the  differences  ob¬ 
served  for  Operational  Checkouts  and  Adjust  activities.  This 
leads  to  the  obvious  conclusion:  that  one  can  reliably  assign  in¬ 
experienced  technicians  with  PIMO  with  the  reasonable  expecta¬ 
tion  that  maintenance  couid  be  accomplished  within  a  very  small 
time  difference  from  that  of  a  qualified  technician  and  with  no 
errors. 

c.  Mann -Whitney  U  Test 

Selected  samples  were  abstracted  from  the  larger  data  base  and 
subjected  to  analysis  by  the  Mann-Whiiney  U  technique.  It  was 
the  purpose  of  this  test  to  provide  an  additional  measure  of 
validity  to  the  data  analysis.  In  no  way  did  the  results  of  this 
analysis  contradict  the  apparent  trends  as  evolved  from  the 
analysis-of-variance.  These  tests  did  show  significance  in  some 
instances  when  PIMO  presentation  modes  were  compared  with 
the  Standard  Operating  Procedures.  The  results  from  this  test, 
together  with  the  high  F-ratios  derived  during  the  analysis-of- 
variance,  allows  a  great  deal  of  reliance  to  be  placed  upon  the 
inferences  drawn. 

In  the  main,  PiMG  produced  lunger  maintenance  times  than  SOP. 
It  should  oe  pointed  out,  however,  that  the  field  test  lasted  for 
only  four  months,  and  that  the  technicians  were  exposed  to  this 
entirely  new  method  of  maintenance  only  for  the  duration  of  this 
test.  There  are  good  indications  described  in  a  later  section 
that  learning  effects  may  have  contributed  to  the  observed  dif¬ 
ferences.  (These  later  analyses  were  made  to  investigate  the 
learning  effects,  and  to  better  describe  and  evaluate  any  ob¬ 
servable  trends).  When  comparing  the  two  PIMO  methods,  the 
results  of  the  test  show  the  booklet  mode  to  be  more  effective 
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than  the  audio-visual  mode.  The  observed  time  differences 
between  the  two  presentation  modes  were  not  evenly  distributed 
across  the  two  groups  of  technicians.  In  fact,  they  were  con¬ 
siderably  smaller  for  apprentices  than  experienced  technicians. 

This  would  seem  to  indicate  that  the  audio-visual  mode  may  pro¬ 
vide  more  help  to  technicians  with  little  experience,  than  to  tech¬ 
nicians  with  a  high  degree  of  experience  and  specialization. 
Moreover,  the  same  observation  seemed  to  be  borne  out  when 
the  two  groups  of  technicians  were  contrasted.  The  experienced 
group,  while  generally  performing  maintenance  in  less  time  than 
the  apprentices,  did  relatively  better  when  they  used  the  books. 
Stated  differently,  the  differences  between  the  two  groups  of 
technicians  v/ere  smaller  with  the  audio-visual  mode  of  data 
presentation  than  with  the  booklet  mode. 

3.  TSA  Evaluation  Study  ' 

Figure  6-3  is  a  bar  graph  which  contrasts  the  mean  performance  times 
for  troubleshooting  with  the  TSA's  and  with  the  standard  troubleshooting 
procedures.  The  data  show  that  the  use  of  the  PIMO  troubleshooting 
manuals  resulted  in  an  11%  decrease  in  troubleshooting  time.  When 
tested  for  significance,  (students  _t),  it  was  shown  that  this  was  a  sta¬ 
tistically  significant  difference.  Only  experienced  technicians  were 
used  as  subjects  in  this  test,  since  the  troubleshooting  aids  were  de¬ 
signed  for  them,  and  it  was  not  possible  to  employ  apprentice  technicians. 
Observed  trouble  shooting  errors  were  grouped  by  type  The  errors 
identified  are  presented  in  Table  6-7.  Errors  were  discovered  in  29.  2% 
of  the  troubleshooting  actions  using  SOF's,  while  they  were  discovered 
in  only  2.5%  of  the  troubleshooting  actions  using  the  PIMO  TSA's. 

Equally  as  significant  is  the  fact  that  12.2%  of  the  troubleshooting  actions 
with  SOP,  as  compared  to  2.  5%  of  the  actions  with  the  TSA's,  identified 
components  to  be  replaced  erroneously.  This  has  a  large  impact,  not 
only  on  the  efficiency  with  which  troubleshooting  may  be  accomplished, 
but  upon  the  spares  consumption  rate,  which  in  turn  reflects  upon  the 
over-all  life  cycle  costs  of  the  larger  system. 
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SOURCE  OF  VARIATION 

SS 

df 

MS 

F  RATIO 

A  (PRESENT.  MODE) 

.0021 

1 

.0021 

1.2353 

B  (SKILL  LEVEL) 

.0021 

1 

.0021 

1.2353 

AB 

0 

1 

0 

0 

WITHIN  CELL 

.  1381 

82 

.0017 

SOURCE  OF  VARIATION 

SS 

df 

MS 

F  RATIO 

A  (PRESENT.  MODE) 

.  0263 

2 

.0132 

8.25* 

WITHIN  CELL 

1 1 H  WM 

_ 51 _ 

Sample  Randomized  Design 


F  RATIOS 

F  (1,80)  =  3.96 
F  (2,55)  =4.02 


Table  6-7.  Summary  of  Analysis -of-Variance  Calculations 

for  Adjust 
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4.  Special-Test 

Mention  was  made  in  Section  V  of  this  volume  of  a  special  test  con¬ 
ducted  at  the  conclusion  of  the  basic  field  test.  Its  primary  objective 
was  to  compare  the  PIMO  presentation  modes  with  standard  Air  Force 
Technical  Orders.  Performance  time  and  errors  were  used  as  the 
dependent  variables.  For  purposes  of  this  test,  a  sub-group  of  ap¬ 
prentice  technicians  was  further  divided  into  two  groups.  The  first 
group  had  had  no  previous  experience  maintaining  the  assigned  system. 

The  second  group  had  had  experience  on  the  assigned  system  using  the 

} 

PIMO  Job  Guides.  Comparisons  between  performances  with  che  PIMO 

t 

Job  Guides  and  Technical  Orders  indicate  that  apprentice  technicians 
were  able  to  perform  maintenance  with  the  tech  orders  only  after  they 

i 

had  repeatedly  performed  that  maintenance,  on  the  same  system,  pre¬ 
viously  using  the  PIMO  Job  Guides.  In  fact,  the  recorded  mean  times 
were  approximately  equal  for  both  cases. 

t 

When  a  comparison  was  made  of  the  apprentice  using  PIMO  with  the 

specialists  using  the  tech  orders,  the  data  indicated  that  the  apprentice 

performed  nearly  as  well  as  the  specialists.  On  the  other  hand,  the 

inexperienced  technicians  who  had  no  experience  with  PIMO,  in  seme 

cases  were  unable  to  complete  the  job  without  assistance.  While  these 

results  could  not  be  tested  statistically  due  to  the  small  sample  size, 

they  nonetheless  demonstrated  a  very  important  point  already  made 

apparent  from  the  main  test:  PIMO  Job  Guides  could  function  as  a 

training  device,  without  subjecting  the  apprentices  to  long  training 

periods.  ■  i 

!  * 

For  purposes  of  this  special  study,  four  types  of  procedural  errors 
were  identified.  These  errors  were:  1)  Type  A  --  maintenance  step 
performed  either  out  of  sequence  or  omitted;  2)  Type  B  --  maintenance 
step  performed  incorrectly;  3)  Type  C  --  indication  called  for  errone¬ 
ously  interpreted  and  not  correctly  obtained;  and  4)  Type  ft  --  mainte- 
nance  action  terminated  due  to  lack  of  experience  or  to  lack  of  infor¬ 
mation.  In  analyzing  these  errors.  Types  A,  B,  and  C  were  combined. 
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and  Type  SI  was  treated  separately.  The  primary  statistical  tests 

2 

employed  in  the  analysis  was  X  .  In  other  instances,  Fisher's  Exact 
Probability  Test  was  used;  both  tests  were  held  to  the  5%  level  of  sig¬ 
nificance.  Table  6-8  presents  the  data  gathered  during  the  special  test, 
and  shows  a  comparison  of  the  apprentice  technicians  who  had  prior 
PIMO  experience,  with  the  apprentice  technicians  who  had  no  prior 
experience. 

Figure  6-10  shows  a  bar  chart  comparing  the  mean  maintenance  times 
for  experienced  technicians  under  four  modes  of  data  presentation: 
audio-visual,  booklet,  PIMO,  tech  orders  and  standard  operating  pro¬ 
cedures.  This  is  for  long,  medium  and  short  opcheck  maintenance 
activities.  Figure  6-11  contrasts  the  maintenance  time  for  experienced 
technicians  using  tech  orders  versus  apprentices.  Figure  6-12  reflects 
the  maintenance  times  for  the  two  groups  of  apprentice  technicians  — 
naive  and  experienced  --  when  using  tech  orders,  and  audio-visual 
or  booklet  modes  of  data  presentation.  When  both  groups  were  assigned 
T.O.  rs,  it  can  be  demonstrated  that  both  the  A,  B,  C  errors  and  the 
SI  errors  were  significantly  greater  for  those  technicians  with  no 
prior  experience.  The  obvious  conclusion  is  that  PIMO  is,  in  fact,  a 
useful  training  aid;  and  that  it  will  enable  technicians  to  perform  mainte¬ 
nance  more  accurately  even  when  PIMO  is  not  used. 

Of  even  greater  importance  is  the  suggestion  that  the  apprentice  tech¬ 
nicians  actually  make  far  fewer  errors  than  the  experienced  technicians 
(see  Table  6-9).  This  should  be  taken  with  caution,  however,  since  the 
difference  cannot  be  demonstrated  to  be  statistically  significant.  It  is 
safe  to  conclude  that  apprentice  technician*  after  several  uses  of  PIMO 
are  able  to  perform  maintenance  as  accurately  as  the  experienced  tech¬ 
nicians  when  both  must  perform  maintenance  using  the  standard  T.O.  's. 
If  the  three  modes  of  data  presentation  are  compared  --  for  the  three 
groups  of  technicians  --it  can  be  seen  that  in  all  cases  significantly 
more  A,  B,  C  errors  are  made  with  the  tech  orders,  than  are  made 
with  either  the  audio-visual  or  the  booklet  mode  of  presentation.  More¬ 
over.  the  apprentice  technician  commits  significantly  more  SI  errors 


6-31 


Vol.  2 


Standard 

Operating 

Procedures 

PIMO 
Trouble- 
Shooting  Aids 

Total  Troubleshooting  Actions 

41 

40 

Troubleshoot,  isolated  problem,  removed 
and  replaced  component,  system  opchecked 
as  bad.  Had  to  retroubleshoot  and  remove 
and  replace  a  different  component  before 
system  opchecked  as  good. 

5 

1 

Troubleshoot,  isolated  problem,  performed 
corrective  action  (non-remove),  system 
opchecked  as  bad.  Had  to  retrouble  shoot 
and  perform  additional  maintenance  on 
different  component  before  system  op¬ 
checked  as  good. 

4 

0 

Troubleshoot  but  could  not  isolate  problem, 
opchecked  and  found  bad;  retroubleshoot 
and  isolated  problem. 

3 

0 

Total  Errors 

• 

12 

1 

Percent  Errors 

29.2% 

2.5% 

Table  6-8.  Troubleshooting  Errors  Identified  During  Field  Test 
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Booklet 


T.O. 


Experienced 

Technicians 

n  =  30 

ABC  =  0 

0  =  0 

n  =  21 

ABC  =  0 

0=0 

n  =  9 

ABC  =  16 

n  =  0 

Apprentice 

technicians  with 

prior  PIMO 

experience 

n  =  39 

ABC  =  0 

I?  =  0 

n  =  12 

ABC  =  0 

0=0 

n  =  11 

ABC  =  13 

0=0 

Apprentice 

technicians  with 

no  prior  experience 

n  =  17 

ABC  =  0 

0  =  0 

n  -  8 

ABC  =  0 

0=0 

n  =  9 

ABC  =  49 

0  =  6 

Table  6~3  Errors  Identified  During  Special  Test 
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Figure  8-12  PIMO  vs.  T.O.  Pilot  Test 
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MAINTENANCE  TIMES  FOR  TWO  GROUPS  OF  APPRENTICE  TECHNICIANS 


under  T.O.  's  than  he  made  with  either  audio-visual  or  booklet  mode. 
These  data  seem  to  suggest  that  almost  regardless  of  experimce  all 
technicians  who  use  the  PIMO  Job  Guides  can  perform  maintenance 
with  equal  accuracy. 

5.  Learning  Effects 

Evidence  from  both  field  and  laboratory  research  indicate  that  in  almost 
every  case  where  motor  skills  are  involved,  learning  does  take  place. 

In  order  to  better  evaluate  the  degree  to  which  learning  may  have  been 
acquired  by  the  subjects  of  this  test,  and  the  subsequent  impact  this 
learning  may  have  had  on  their  over-all  performance,  it  was  decided 
to  develop  learning  curves  from  the  collected  data.  The  learning 
curves  were  developed  for  both  experienced  and  inexperienced  person¬ 
nel,  and  are  expressed  as  a  percent  of  the  average  SOP  time.  The 
data  are  presenter!  by  the  number  of  successive  actions  on  the  same 
activity  —  first  time  performed,  second,  third,  etc.  Two  curves  are 
presented;  one  shows  the  data  as  it  was  computed  directly.  This 
curve,  although  showing  a  classical  learning  situation,  is  somewhat 
erratic  due  to  the  random  variation  in  data.  The  second  is  derived 
from  the  first,  and  represents  a  smoother  version.  There  is  good 
reason  to  believe  that  this  curve  is  more  congruent  to  one  which  depicts 
the  actual  learning  process.  These  curves  are  shown  in  Figures  6-13 
and  6-14.  Inspection  of  these  curves  shows  that  there  did  exist  a  definite 
learning  effect  when  the  PIMO  Job  Guides  were  employed.  For  example, 
the  first  action  by  an  experienced  technician  using  PIMO  took  approxi¬ 
mately  2.8  times  as  long  as  it  did  an  experienced  technician  using  his 
normal  means  of  maintenance.  However,  his  performance  time  drop¬ 
ped  rapidly  until  at  the  end  of  the  eighth  performance  he  was  performing 
the  maintenance  using  PIMO  in  less  time  than  he  took  using  SOP.  In  a 
like  fashion,  the  first  action  by  inexperienced  technicians  was  shown  to 
take  approximately  2.  5  times  as  long  as  an  experienced  technician  using 
his  Standard  Operating  Procedures,  Here  too,  the  performance  time 
pi  ogiessively  decreased  until  approximately  the  twelfth  repetition 
where  it  was  shown  to  take  the  inexperienced  technician  approximately 
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Figure  6-13  Learning  Curve 
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SUCCESSIVE  MAINTENANCE  ACTION  NUMBER 


Figure  6-14  Smoothed  Learning  Curve 


i.  2  times  as  long  as  the  experienced.  While  no  data  are  available  past 
this  point,  it  is,  nonetheless,  believed  that  TIF  continues  to  decrease 
with  experience,  until  such  time  as  the  inexperienced  technician  per¬ 
forms  at  a  rate  of  the  experienced  technician. 

In  effect,  there  are  really  two  learning  effects  coming  into  play  here. 
One  is  the  learning  process  associated  with  the  use  of  PIMO  materials 
and  the  second  is  the  learning  effect  as  a  consequence  of  repeating 
maintenance  many  times  on  specific  hardware.  The  experienced  tech¬ 
nicians,  we  can  safely  assume,  were  already  trained  on  the  hardware 
and  were  performing  maintenance  which  indicated  only  the  learning 
effect  due  to  PIMO.  On  the  other  hand,  the  inexperienced  personnel 
untrained  in  the  particular  system,  reflect  in  their  learning  curves 
both  effects  --  the  learning  of  PIMO,  and  the  learning  of  hardware. 


3 .  Time  Change  Matrix 

The  time  change  matrix  (see  Figure  6-15)  reflects  for  each  pair  of 
maintenance  actions  --  for  identical  man  and  activity  --  the  average 
percentage  of  time  change  by  mode.  The  matrix  does  not  give  values 
which  would  be  capable  of  developing  a  learning  curve;  rather  it  intends 
to  assess  the  relative  learning  between  the  various  modes  and  experi¬ 
ence  levels,  and  to  add  validity  to  the  learning  carves  already  develop¬ 
ed.  Two  matrices  are  presented:  one  for  the  experienced,  one  for 
the  inexperienced  technician.  They  are  presented  in  the  form  of  a 
3x3  and  a  2  x  2  contingency  table  for  the  experienced  and  inexperi¬ 
enced  technicians,  respectively.  For  both  technicians  the  percentage 
change  from  booklet  to  booklet  is  greater  than  that  from  audio-visual 
to  audio-visual.  This  would  seem  to  suggest  that  learning  may  be 
faster  with  the  booklets  than  with  the  audio-visual.  However,  since  i 
was  mentioned  previously  that  there  were  two  learning  effects  coming 
into  play,  it  is  impossible  to  identify  the  contribution  of  each  to  the 
percentage  change.  Therefore,  even  though  learning  with  the  booklet 
may  appear  to  be  faster,  it  is  not  known  if  the  learning  of  the  PIMO 
methods,  the  hardware,  or  both  is  really  the  major  contributor. 


The  data  also  seem  to  indicate  that  learning  for  the  experienced  tech¬ 
nicians  is  greater  than  that  for  the  inexperienced  technicians.  This 
would  follow  logically,  since  it  can  be  assumed  that  the  differences  in 
the  learning  rates  are  due  in  part  to  the  learning  of  the  hardware  by 
the  inexperienced  technicians. 
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Figure  6-1j.  Time  Change  Matrix 
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Figure  6-16.  Impact  of  PIMO 
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D.  USER  ACCEPTANCE 


In  order  to  determine  the  reaction  of  the  user  toward  PIMO  Job  Guides 
and  TSA  manuals  an  attitude  questionnaire  was  developed.  This 
questionnaire  was  administered  to  all  subjects  twice  during  the  course 
of  the  field  test.  The  first  administration  occurred  at  the  halfway 
point,  when  presentation  modes  were  changed.  This  questionnaire, 
slightly  modified  to  account  for  the  differences  in  experience,  was 
administered  again  at  the  conclusion  of  the  field  test. 

Originally,  it  had  been  planned  to  analyze  the  data  through  the  use  of 

2 

non-parameteric  statistics,  such  as  x  .  Inspection  of  the  data  revealed 
that  it  was  distributed  in  ratios  so  large  that  it  made  inferential  statis¬ 
tical  analysis  unnecessary.  The  questionnaire  items  are  separated 
into  two  broad  categories.  The  first  is  that  of  the  subject's  demographic 
data,  which  recorded  his  age,  skill  level,  length  of  time  in  service, 
etc.  The  second  broad  category  is  that  of  attitudes  of  PIMO  acceptance, 
and  is  composed  of  both  multiple  choice  attitude  questions  and  open- 
ended  questions.  The  questionnaire  was  administered  under  rigidly 
controlled  circumstances  in  the  field,  and  it  is  believed  that  the  data 
gathered  from  this  questionnaire  is  highly  reliable.  *  Each  mainte¬ 
nance  technician  at  the  conclusion  of  the  test  had  used  both  the  PIMO 
modes  of  presentation  a  large  number  of  times.  Therefore,  the 
questionnaire  responses  should  have  been  based  upon  intimate  familiar¬ 
ity  with  the  PIMO  methods. 

Table  6-10  shows  the  demographic  characteristics  of  all  PIMO  subjects. 
The  median  age  for  the  subjects  was  twenty-three  years.  Sixty-eight 
and  nine-tenths  percent  of  the  technicians  were  5 -levels,  and  14  percent 
of  the  technicians  were  3 -levels.  This  left  a  remainder  of  17  percent 
for  7-  and  9-levels,  The  length  of  time  in  seivice  for  C-141A  main¬ 
tenance  tended  to  distribute  itself  bimodally.  Over  59  percent  of  the 
technicians  had  been  in  the  Air  Force  for  more  than  a  year,  while 


While  teste  of  this  nature  are  highly  reliable,  their  validity  may  not 
be  great  as  quantitative  tests. 
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Age  of  Technicians 


16  -  20  years 
2.'  -  25  years 
26  -  30  years 
31  -  35  years 
36  and  over 


17.  3% 

48.  7% 

10.  5%  Median  age  -  23  years 

10.  2% 

13.3% 


b.  Skill  level  of  Technicians 

3 -level 

14.  1% 

5 -level 

68.9% 

7-  and  9-levels 

17. 1% 

c.  Length  of  time  technician  has  been  maintaining  C-141A's 

1-6  months  20.  9% 

7-12  months  19.8% 

Over  12  months  59.  3% 


d.  Education 

1)  Experienced  Technicians 

33  percent  had  some  college  while  the  remainder  had 
finished  high  school. 

2)  Inexperienced  Technicians 

20  percent  had  some  college,  13  percent  had  some 
trade  school,  the  remainder  had  finished  high  school. 


Table  6-10  Demographic  Characteristics  of  Subjects 


Si 


over  20  percent  had  been  in  service  less  than  6  months.  This  distri¬ 
bution  in  no  way  was  allowed  to  color  the  data  taken  during  the  basic 
test,  since  a  balanced  design  was  used. 

One  of  the  significant  questions  asked  in  this  questionnaire  was  whether 
or  not  the  technicians  considered  the  PIMO  aids  useful.  The  responses 
to  these  questions  are  presented  on  Table  6-11,  and  are  listed  according 
to  length  of  time  the  technicians  have  been  performing  maintenance  on 
the  C-141  aircraft.  In  addition,  the  question  is  broken  down  into  the 
various  characteristics  of  PIMO,  such  as  troubleshooting  aids,  job 
guide  index,  input  conditions,  etc. 

Almost  irrespective  of  length  of  time  in  service,  the  troubleshooting 
aids  were  rated  as  useful.  It  seems  w  follow  a  trend  that  the  less  time 
in  service  the  technician  had,  the  more  useful  he  thought  it  was. 
Eighty-two  percent  of  those  technicians  in  service  6  months  or  less 
specified  that  they  felt  the  troubleshooting  aids  were  useful.  Only 
72-1/2  percent  of  technicians  in  service  over  a  year  felt  that  the 
troubleshooting  aids  were  useful.  Nonetheless,  this  does  represent 
a  significant  number  of  technicians.  Approximately  three -fourths  of 
them  felt  that  the  troubleshooting  aids  were  useful. 

Maintenance  instructions  also  rated  extremely  high.  Here,  over  84 
percent  of  the  technicians  in  service  less  than  6  months,  and  almost 
94  percent  of  the  technicians  in  service  between  S  months  and  a  year, 
rated  the  maintenance  instructions  useful.  The  only  attribute  of  the 
PIMO  format  which  fell  below  50  percent,  was  "Input  Conditions". 

This  would  seem  to  say  that  the  input  conditions  were  felt  to  be  not 
as  useful  as  the  other  characteristics.  But,  half  of  any  given  number 
of  maintenance  technicians  still  represents  a  large  part  of  the  mainte¬ 
nance  force. 


In  all  cases,  the  technicians  in  service  on  C-141A's  for  over  12  months 
rated  the  PIMO  aids  as  somewhat  less  useful  than  those  technicians  in 
service  for  less  than  12  months.  This  is  not  surprising  since  as  the 
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Question 

Length  of 
Maintenance 

Useial 

Not  Useful 

1-6  months 

82.0  % 

18.0% 

Troubleshooting  Aids 

7-12 

79.5 

20.5 

over  12 

72.4 

27.6 

Total 

75.3 

24.2 

1-6  months 

70.9% 

29. 1% 

Job  Guide  Index 

7-12 

77.2 

22.3 

• 

over  12 

65.8 

34.2 

Total 

69.1 

30.9 

1-6  months 

44.  C% 

56,0% 

Input  Conditions 

7-12 

51.0 

49.0 

over  12 

43.2 

56.3 

Total 

44.8 

55.2 

1-6  months 

64.3% 

35.7% 

Prepare  for  Activity 

V  -  12 

59.1 

40.9 

over  12 

48.5 

51.5 

Total 

52.4 

47.6 

1-6  months 

84.6% 

15.4 

Maintenance  Instructions 

7  -  12 

93.8 

6.2 

over  12 

73.8 

26.2 

Total 

78.5  - 

21.5 

Table  6-11  Usefulness  of  PIMO  Response 
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experience  level  increases,  the  usefulness  of  any  maintenance  aid 
decreases.  It  is  interesting  to  note,  however,  that  the  technicians 
in  service  between  7  and  12  months  rated  three  out  of  the  five  PIMO 
characteristics  as  more  useful  than  those  in  service  for  less  time. 

One  reason  offered,  is  that  the  newer  rated  C-141A  technicians  are 
probably  3 -levels  who  have  a  limited  experience  in  performing  main¬ 
tenance  on  that  aircraft.  Therefore,  their  use  of  PIMO  was  as  part 
Cu,  or  in  lieu  of,  an  OJT  training  environment. 

Several  questions  were  designed  to  obtain  the  technician's  reactions 
to  statements  concerning  PIMO  aids.  The  responses  to  these  questions 
were  converted  to  a  nominal  scalet  assuming  equal  intervals  between 
the  categories.  The  results  were  than  averaged  and  plotted  graphically 
as  shown  in  Table  6-12.  In  general,  it  can  be  seen  that  the  summary 
of  responses  all  reside  on  the  positive  side  of  the  scale.  In  two  cases 
—  question,  "I  do  not  do  a  better  maintenance  job  using  PIMO  aids", 
and  question,  "Maintenance  can  be  done  faster  using  PIMO  aids"  -- 
the  responses  were  clearly  undifferentiated  and  constellate  around  the 
"don't-know"  portion  of  the  scale.  In  no  case  do  the  subjects  indicate 
-  that  they  strongly  disagree.  One  can  conclude  from  this  that  there 
would  be  very  little,  if  any,  resistance  to  the  use  of  these  aids  were 
they  implemented  as  an  operational  system.  As  discussed  earlier  in 
this  report,  simply  having  a  new  and  more  efficient  technique  for 
formatting  technical  data  is  not  sufficient  to  introduce  positive  changes 
in  over-all  system  effectiveness.  It  goes  without  saymg  that  the  tech¬ 
nicians  m  st  us**  the  data  in  order  for  its  effectiveness  to  be  felt. 

One  item  in  the  questionnaire  asked  for  responses  concerning  whether 
or  not  the  technicians  felt  the  Air  Force  was  heading  in  the  right 
direction  oy  developing  PIMO.  The  results  of  this  question  are  shown 
in  Table  6-13.  The  responses  to  this  question  were  favorable,  with 
almos  70  percent  of  the  technicians  responding  that  PIMO  was  a  step 
in  the  right  direction.  A  gam.  it  should  be  noted,  that  as  the  experience 
level  increased,  the  positive  attitude  towards  PIMO  decreased.  This 
is  not  to  say,  however,  that  the  attitude  ever  became  negative,  since 


5 


Agree  Agree  Don't  Disagree  Disagree 

Strongly  Mildly  Know  Mildly  Strongly 


P1MO  Aids  are  easy  to 
use. 


PIMO  Aids  should  be 
available  for  other  air¬ 
craft  as  well  as  C-141A. 


I  do  a  better  maintenance 
job  using  PIMO  Aids. 


Maintenance  can  be  done 
faster  using  PIMO  Aids. 


The  PIMO  Aids  are  good 
training  aids. 


PIMO  Job  Guides  are 
good  reference  material. 


The  multi- man  task  des¬ 
criptions  are  helpful. 


Table  6-12.  Technicians  reactions  to  PIMO  job  guides. 


'  Length  of 
Maintenance 

Right 

Direction 

Wrong 

Direction 

1-6  months 

83.  5% 

16.5% 

7-12  months 

68.2% 

31.3% 

Over  12  months 

65.5% 

34.4% 

Total 

69.  5% 

30.  5% 

Table  6-13  Direction  of  the  Air  Force  with  PIMO 
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almost  66  percent  of  the  technicians  with  oyer  12  months  of  service 
responded  that  they  felt  the  FIMO  technique  was  a  step  in  the  right 
direction. 

The  technicians  were  also  requested  to  state  their  likes  and  dislikes 
of  the  PIMO  booklets.  Each  respondent  was  allowed  to  indicate  a 
maximum  of  five  likes  and  dislikes.  The  number  of  responses  from 
these  items  were  totaled,  averaged,  and  are  presented  in  Table  6-14. 
Several  characteristics  of  the  formats  were  provided  as  dimensions: 
for  example,  size,  weight,  level  of  detail,  organization,  language, 
index,  etc.  One  interesting  note  is  that  the  technicians  tended  to 
disagree  on  their  responses.  For  instance,  for  the  "likes",  the  tech¬ 
nicians  gave  an  average  of  2.  73  responses;  but  only  slightly  above 
half  of  the  respondents  were  positive  about  the  size,  which  was  the 
highest  ranking  "like".  On  the  other  hand,  for  the  "dislikes"  the 
responses  were  more  uniform.  The  highest  ranking  "dislike"  received 
a  response  from  slightly  over  one-fourth  of  the  respondents.  Finally, 
it  should  be  noted  that  the  technicians  indicated  77  percent  more  "likes" 
than  "dislikes". 

Subsequent  to  the  completion  of  the  field  test,  the  second  questionnaire 
was  administered.  In  this  questionnaire,  the  technicians  were  asked 
which  mode  of  presentation,  if  any,  they  would  use  for  maintenance 
jobs  which  occurred  at  different  frequencies.  The  results  of  this 
question  are  presented  in  Table  6-15.  Using  the  binomial  test  of  sig¬ 
nificance  at  the  .05  level  of  significance,  it  was  determined  that  there 
was  a  preference  for  the  PIMO  mode  of  presentation  by  both  skill 
levels,  for  all  jobs  which  occurred  with  infrequent  regularity.  Sur¬ 
prisingly,  approximately  one-half  of  the  technicians  indicated  a  prefer¬ 
ence  for  PIMO  Guides  for  those  jobs  which  occur  with  greater  regularity. 

Given  that  the  PIMO  method  of  presentation  was  preferred,  the  skilled 
technicians  indicated  a  preference  for  the  booklets  over  the  audio-visual 
device  for  those  tasks  which  occur  frequently  or  even  occa  jionally. 

For  tasks  which  occur  very  seldom,  there  was  no  preferential  difference 


Characteristic 

Like 

Dislike 

Size 

.54 

.09 

Weight 

.30 

1 

.05 

Level  of  Detail 

00 

*■4 

• 

.22 

Technical  Accuracy 

.15 

.23 

Illustrations 

.43 

.11 

Organization 

.24 

.18 

Job  Preparation  Information 

.16 

.18 

Language 

o 

CM 

• 

-.11 

Ease  of  Use 

.38 

.08 

Index 

.13 

.18 

Other 

.02 

.06 

Tccal 

2.73 

1.54 

Table  6-14  Technician's  Opinion  of  FIMO  Characteristics 


Either 


Neither 


Frequency  Group  |Vi£mal 


xpenence 


Booklets  I  Presentation  I  Presentation 


1  -  a  jo 
done  very 
frequently 


3  -  a  job 

seldom 

done 


4  -  a  job 
never  done 


112 


Inexperi¬ 

enced 


Experienced 


Inexperi¬ 

enced 


Experienced 


Inexperi¬ 

enced 


Experienced 


Inexperi¬ 

enced 


Table  6-15 
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between  the  booklets  and  the  audio-visual  modes.  For  unskilled  tech¬ 
nicians,  on  the  other  hand,  there  was  a  significant  preference  for  the 
booklets  over  the  audio-visual  device  for  all  tasks,  regardless  of  their 
frequency  of  occurrence.  This  tends  to  support  the  other  evidence, 
that  the  technicians  who  have  used  the  PIMO  method  of  presentation 
do  prefer  and  would  use  that  mode  when  it  was  available.  It  also  tends 
to  indicate  that,  in  general,  there  is  a  preference  for  the  booklet  over 
the  audio-visual  device.  Here  again,  this  result  falls  in  line  with  the 
conclusions  and  recommendations  for  the  Operational  System  as  out¬ 
lined  in  Volume  III.  As  noted  earlier,  a)  since  there  were  little  or  no 
significant  differences  in  performance  times  for  the  A-V  versus  the 
booklet  mode  of  presentation,  b)  and  since  technicians  do  seem  to 
indicate  a  preference  for  booklets,  c)  and  further,  since  the  booklet 
mode  represents  a  less  expensive  system  to  initiate  and  to  operate, 
the  booklet  modes  evolve  as  the  recommended  technique. 

Other  questions  on  the  second  questionnaire  asked  the  technicians  to 
state  their  agreement/disagreement  with  several  statements  concerning 
the  possible  impact  of  the  PIMO  aids.  Such  questions  as,  "The  PIMO 
aids  are  easy  to  use",  "I  do  a  better  maintenance  job  using  PIMO", 

"The  PIMO  aids  are  good  training  aids",  were  included.  Graphical 
results  of  the  questions  were  obtained  by  applying  a  numerical  value 
to  each  possible  response,  with  the  underlying  assumption  that  equal 
intervals  between  the  responses  existed.  The  values  obtained  in  this 
manner  were  then  averaged  and  plotted  graphically.  A  statistical 
analysis  was  performed  which  used  only  the  direction  of  the  responses 
without  regard  to  magnitude.  The  tests  used  were  the  Fisher  Exact 
Probability  Tests  and  the  Binomial  Test  of  Significance  --  both  at  the 
.05  level  of  significance.  The  results  of  the  responses  to  these  questions 
are  plotted  in  Figure  6-15.  Included  are  the  responses  for  both  modes 
of  presentation,  and  for  both  groups,  (i.  e.,  skilled  and  unskilled).  It 
can  be  seen  that  the  unskilled  technicians  as  a  group  gave  a  more 
favorable  response  than  the  skilled  technicians  --  in  all  cases  save 
three.  Of  these  three  cases,  twc  responses  were  identical  to  the 


skilled  personnel,  and  one  was  in  a  more  unfavorable  direction.  These 
data  disclose  that  the  unskilled  personnel  significantly  recognized  the 
usefulness  of  PIMO  than  do  the  skilled  personnel.  Moreover,  the  data 
show  that  both  groups  gave  a  favorable  response  significantly  more 
often  than  an  unfavorable  response.  If  these  six  questions  can  be 
considered  as  approximate  descriptors  of  the  PIMO  concepf  in  general, 
then  one  can  infer  that  both  levels  of  personnel  are  significantly  favor¬ 
able  in  their  attitude  towards  the  concept;  and  that  in  addition,  the 
unskilled  personnel  are  significantly  more  favorable  than  are  the  skilled 
personnel.  Here  again,  the  responses  to  the  booklet  were  in  a  direction 
more  favorable  than  the  responses  to  the  audio-visual.  This  supports 
the  data  taken  from  the  earlier  questions  and  the  discussion  presented 
earlier  in  this  volume.  Last,  it  should  be  noted  that  the  questions 
which  received  the  most  favorable  responses  were  those  concerned 
with  PIMO  as  a  training  aid  and  as  reference  material.  Both  groups 
of  subjects  showed  strong  agreement  to  these  statements,  but  seemed 
to  prefer,  in  both  instances,  the  booklet  over  the  A-V  presentation  mode. 

Since  two  questionnaires  v/ere  issued,  it  was  possible  to  perform  an 
analysis  to  determine  if  the  technicians'  attitudes  towards  the  PIMO 
concept  changed  with  time.  All  questionnaires  administered  at  the 
halfway  point  were  matched  to  the  final  questionnaire  by  the  identical 
respondent.  The  differences  in  the  responses  between  the  two  question¬ 
naires  to  key  questions  were  recorded  and  analyzed  {questions  1  through 
6  of  Section  IV  were  the  key  selected  questions).  It  was  found  that 
responses  to  the  questions  on  the  final  questionnaire  were  significantly 
more  favorable  than  were  the  questions  by  the  identical  technicians  on 
the  first  questionnaire.  This  would  indicate  that  the  user's  attitude 
towards  the  questionnaire  became  more  favorable  as  they  became  more 
familiar  with  the  concept,  through  use. 

In  re -questioning  the  subjects  concerning  the  usefulness  of  the  PIMO 
aids,  a  shift  was  indicated  towards  the  useful  side.  Table  6-16  shows 
that  for  the  inexperienced  technicians,  100  percent  agreed  that  the  job 
guides  were  useful  for  all  maintenance  activities  except  troubleshooting. 
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QUESTION 

EXPERIENCED 

TECHNICIANS 

INEXPERIENCED 

TECHNICIANS 

Useful 

Not 

Useful 

Useful 

Not 

Useful 

Troubleshooting  Aids 

100% 

0% 

N/A 

N/A 

Job  Guides  --  for 

Remove /install 

73% 

27% 

100% 

0% 

Job  Guides  --  for 

Checkout 

91% 

9% 

100% 

0% 

Job  Guides  --  for 

Adjust 

100% 

0% 

100% 

0% 

Table  6-16  Before  and  After  Responses 


Here,  of  course,  the  question  was  not  applicable  since  inexperienced 
technicians  were  not  used  as  subjects  in  the  troubles  hootinp  aids 
effectiveness  study.  On  the  other  hand,  the  experienced  technicians 
agreed  100  percent,  that  the  troubleshooting  aids  were  useful;  and 
except  for  Remove  and  Install,  agreed  almost  100  percent  that  the  job 
guides  were  useful.  (For  Remove  and  Install,  73  percent  agreed  that 
they  were  useful). 

When  the  technicians  were  asked  again  if  they  felt  that  PIMO  was  a 
step  in  the  right  direction  on  the  part  of  the  Air  Force,  the  responses 
were  as  follows:  80  percent  of  the  experienced  group  agreed  and  100 
percent  of  the  inexperienced  group  agreed  that  this  was  the  right  step. 
One  can  conclude  from  this,  that  not  only  do  the  technicians  feel  PIMO 
Job  Guides  --  as  they  have  experienced  them  --  were  useful  in  a  per¬ 
sonal  sense,  but  that  applied  to  all  Air  Force  aircraft  systems,  the 
job  guides  would  provide  a  substantial  support  in  maintenance  activities. 

The  technicians  were  again  given  an  opportunity  to  rate  their  likes  and 
dislikes  of  certain  characteristics  of  the  PIMO  format.  These  charac¬ 
teristics  included  size,  weight,  level  of  detail,  language,  illustration 
and  organization,  as  examples.  The  questions  were  weighted  according 
to  response  and  the  values  were  averaged  for  each  group  of  technicians. 
The  results  of  this  analysis  are  shown  in  Table  6-16.  The  higher  the 
number,  the  greater  the  response  in  the  direction  of  "like".  Of  par¬ 
ticular  interest  is  the  comparison  between  the  responses  of  the  un¬ 
skilled  technicians  as  to  what  they  liked  least  about  the  audio-visual 
and  booklet  size.  Table  6-17  shows  that  size  was  least  liked  about  the' 
audio-visual  presentation  for  unskilled  technicians.  The  third  and 
fourth  highest  values  for  both  levels  of  technician  was  booklet  size. 
Stated  more  simply,  this  shows  that  the  size  of  the  booklet  was  one  of 
the  better  features  for  its  mode,  v’hile  the  size  of  the  A-V  device  was 
one  of  the  poorer  features  for  its  mode. 

Lastly,  a  section  of  the  questionnaire  contained  open-ended  questions 
for  completion  by  the  subjects.  It  was  felt  that  this  technique  would 
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♦  Generally,  the  inability  to  repeat  individual  frames  in  the  A-V  device. 


provide  valuable  insight  into  aspects  of  the  user's  feeling  of  acceptance 
or  rejection  that  was  heretofore  untapped.  Since  questions  of  this  type 
are  difficult  to  categorize  and  to  summarize,  it  was  felt  that  selected 
responses  typical  of  those  received  should  be  presented  wholly  as  they 
were  taken  from  the  questionnaire  itself.  They  are  loosely  organized 
by  skill  level. 

1.  Skilled  Personnel 

"I  think  it  would  be  good  in  a  training  program. " 

"PIMO  is  no  better  or  worse  than  the  present  T.O.  system.  If, 
as  in  the  case  of  hydraulic  systems,  the  program  had  not  had  so 
many  obvious  errors  it  would  have  been  better  received.  " 

"I  feel  the  program  was  very  helpful  and  interesting.  The  company 
representatives  were  capable,  willing  and  I  feel  they  did  themselves 
and  the  Air  Force  a  job  well  done". 

"PIMO  is  a  very  good  aid  for  a  unskilled  man  (Booklet  &  A-V), 
a  skilled  man  profits  more  from  the  troubleshooting  aids  than 
A-V  or  booklet.  If  I  ran  into  position  where  I  needed  a  reference 
I  would  like  to  be  able  to  have  a  booklet  instead  of  a  tech  order." 

"It  was  good  for  training  purposes  but  A-V  was  too  slew  and 
bulky. " 

"I  feel  that  PIMO  aids,  such  as  the  organization  troubleshooting 
aids  and  booklets,  can  be  used  to  increase  maintenance  efficientoy. 
It  is  also  an  aid  in  3  level  training.  I  only  wish  that  we  could  have 
been  able  to  use  the  aids,  in  our  more  complex  sys. " 

"Illustrations  could  be  more  clear. " 

"The  guides  are  written  in  a  maintenance  mans  language  and  is 
easily  understandable. " 

"I  believe  PIMO  aids  should  definitely  be  used  for  training  purposes 
and  for  jobs  a  specialist  may  never  have  done  before.  " 
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"I  like  the  PIMO  booklets.  T  think  the  A/V  units  are  not  as  con¬ 
venient' to  use  as  the  bookb.  The  A/V  units  would  be  more  helpful 
in  training  than  actual  working  procedures.  " 

"PIMO  would.be  a  very  good  training  aid  for  the  unexperienced 
mechanic  in  a  specific  field." 

"PIMO  is  adequately  reduced  in  size  for  the  booklets  but  a  useable 
A/V  device  should  be  smaller  with  repeat  capabilities  for  sound 
and  video.  The  aids  are  approaching  the  solution  to  bulky,  poorly 
written,  difficult  to  follow  technical  orders  and  utilization  of  a 
system  such  as  PIMO  is,  in  my  opinion,  a  definite  improvement 
over  prevailing  system. " 

There  were  three  responses  of  "No  comment." 

2.  Unskilled  Personnel 

"The  booklets  are  very  handy  light  weight  and  easy  to  use.  The 
detail  or  technical  accuracy  of  a  few  jobs  needs  revising.  The 
A/V's  are  to  heavy  and  cumbersome  and  there  is  no  way  of 
backing  up  a  frame.  This  is  much  needed.  Besides  these  few 
things  I  think  PIMO  is  a  real  good  system." 

"That  the  PIMO  aids  are  easy  to  understand  good  to  use  for 
maintenance. " 

"Audio-vis.  aids  are  slightly  bulky  to  carry  for  a  long  distance 
and  are  incapable  of  instent  replay.  The  booklet  is  easy  to  under¬ 
stand  and  work  with.  I  feel  that  the  PIMO  aids  are  excellent  for 
maintenance  but  for  training  in  over  all  systems  training,  they  do 
not  present  enough  information  to  get  through  their  tests  in 
training. " 

"The  PIMO  aids  do  not  show  some  important  items  that  should 
shown  to  people  who  are  not  familiar  with  a  certain  system.  I 
think  the  booklets  are  better  than  Audio-Visual  because  they  are 
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easier  to  carry  and  easier  to  understand.  " 

"They  are  easier  to  use  than  T.O.  's.  They  are  also  lighter  and 
easier  to  handle." 

"PIMO  booklets  seem  to  be  very  good  training  devices  and  are 
very  easy  to  use.  I  think  A  /V  device  would  also  make  a  good 
training  aid,  but  to  me  seem  much  harder  to  use  than  the  booklet. " 

"I  think  the  booklets  are  almost  perfect  except  some  of  the  wording 
is  wrong  or  misleading.  Either  booklet  or  A  /V  would  be  an  excel- 
lent  training  aid  for  tech  school  or  upgrading  to  5  level.  " 

"They  were  easy  to  understand.  For  this  3  level  they  were  good 
training  aids.  I  did  not  know  any  thang  about  141  's  before  I  cam 
into  PIMO.  PIMO  inabled  me  to  perform  actions  which  I  could 
not  otherwise  do." 

"PIMO  aids  are  very  useful.  Everything  very  clear." 

"The  booklets  are  good  and  need  little  improvement,  if  any.  The 
A  /V  are  good  but  need  a  good  deal  of  refinement.  They  are  too 
large  and  in  some  cases  the  verbile  task  is  too  long  for  the 
unskilled  man  to  grasp,  " 

There  were  five  responses  of  "No  comment". 

In  summary,  it  is  fairly  obvious  that  the  technicians  were  positively 
inclined  toward  the  PIMO  format.  This  was  true  at  least  by  propor¬ 
tions  significantly  greater  than  half  of  those  who  were  users.  In 
almost  no  instance  was  there  a  negative  reaction  to  the  guides.  If 
the  results  of  these  questionnaires  can  be  accepted  as  valid  --  the 
reliability  has  certainly  been  demonstrated  in  the  matching  responses 
of  the  two  applications  of  this  questionnaire  --  then  one  can  safely 
assume  that  given  the  opportunity,  a  maintenance  technician  would 
need  little  or  no  encouragement  to  employ  PIMO.  This  is  extremely 
important  when  considered  in  light  of  the  empirically  observed  use 
of  standard  tech  data.  In  only  about  five  percent  of  the  cases  do 
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technicians  currency  employ  the  standard  technical  orders.  This  is 
not  to  say  that  PIMO  would  be  used,  if  it  were  available,  to  any  larger 
degree.  But  certainly  if  the  over-all  maintenance  system  were  to 
change  in  the  direction  as  suggested  in  Volume  III  of  this  report  --  that 

is,  toward  the  employment  of  lesser  skill  levels  --  then  it  becomes 
important  to  have  technical  orders  which  not  only  can  provide  the  infor¬ 
mation  in  a  useful  manner,  but  in  a  way  which  is  found  acceptable  by 
the  user  himself. 

Stated  in  another  way,  it  makes  little  rense  to  employ  lower  skill  level 
technicians  to  perform  maintenance,  giv  jn  supporting  tech  data,  if, 
he  is  reluctant  or  resistant  to  the  use  of  that  tech  data.  It  hat:  been 
said,  that  if  pressure  were  applied  to  the  technician,  he  would  tploy 
tech  data.  This  certainly  is  open  to  question.  It  is  true  that  one  might, 
through  insistence,  achieve  the  state  wherein  technicians  took  the  tech 
data  with  them  to  the  job  site.  But  this  is  not  to  ensure  that  the  data 
would  be  consulted  by  the  technician.  To  paraphrase  an  old  proverb, 
"You  can  lead  a  technician  to  data,  but  you  can't  always  make  him  use 

it. "  Responses  to  the  open  ended  questions  seem  to  indicate  a  genuine 
like  for  the  technical  data.  For  instance,  one  technician  wrote  that 
he  felt  that  the  guides  were  written  in  a  maintenance  man's  language 
and  were  easily  understood.  Another  technician  commented  that  he 
felt  the  PIMO  Job  Guides  were  easier  to  use  than  the  standard  tech 
orders  and  were  both  lighter  and  easier  to  handle.  These  responses 
are  not  surprising,  in  light  of  the  effort  that  went  into  the  generation 
of  a  glossary  of  terms  derived  directly  from  the  field  maintenance 
technicians.  Finally,  if  the  Air  Force  decides  to  implement  the  PIMO 
formatting  on  a  system-wide  or  Air  Force -wide  basis,  there  appears 
tc  be  little  or  no  problem  in  generating  acceptance  of  these  new  tech 
ordero  on  the  part  of  the  maintenance  technician. 
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SYSTEM  EFFECTIVENESS  RESULTS 

The  field  test  indicated  that  the  PIMO  technique  of  data  presentation 
will,  in  fact,  produce  a  positive  change  in  C-141  maintenance  proce¬ 
dures.  The  observed  effect  of  PEMO  on  the  maintenance  parameters 
was:  1)  PIMO  will  reduce  the  maintenance  error  rate,  2)  PIMO  will 
enable  lesser  skilled  maintenance  technicians  to  perform  those  func  - 
tions  formerly  performed  by  more  skilled  technicians,  and  3)  PIMO 
may  increase  the  time -in-function,  particularly  for  the  inexperienced 
technician.  This  last  parameter  will  cause  a  negative  change  in  ef¬ 
fectiveness  measures  since  it  would  take  longer  to  perform  maintenance 
functions.  This  is  true  only  with  the  data  as  they  were  presently  ob¬ 
served,  but,  as  mentioned  so  many  times  earlier,  these  data  repre¬ 
sent  the  mean  times  for  all  performances,  including  the  performance 
which  occurs  very  early  in  the  training /learning  cycle. 

The  first  effect  will  beneficially  impact  the  over-all  systems  effective¬ 
ness,  because  fewer  errors  will  result:  a)  in  fewer  reworks  of  the 
maintenance  for  malfunction  correction,  and  b)  fewer  instances  in 
which  malfunctioning  aircraft  are  discovered  in  preflight  inspections, 
either  causing  mission  delay  or  requiring  aircraft  substitution.  The 
second  effect  will  be  beneficial  since  it  will  provide  a  greater  pool  of 
personnel  from  which  maintenance  technicians  may  be  drawn.  To  fully 
understand  the  impact  of  these  parameters  upon  the  over-all  systems 
effectiveness,  it  was  decided  to  exercise  the  AMES  model  and  review 
the  results.  Data  supplied  to  the  model  came  from,  in  most  cases, 
the  PIMO  field  test.  Time -in-function  maintenance  data,  for  example, 
was  derived  in  this  way.  The  time -in-function  useo.  for  the  nominal 
case  was  that  observed  for  the  skilled  technicians  when  using  SOP's. 

The  time  in-function  data  for  the  PIMO  AMES  runs  were  derived  from 
the  unskilled  technician  using  PIMO  data.  It  was  assumed  that  shop 
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repair  functions  would  not  be  affected  by  PIMO,  and  in  this  way  the 
same  skilled  personnel  and  the  same  time -in-function  were  used  in 
both  the  nominal  and  PIMO  run  for  shop  repair  functions.  The  specific 
data  input  were  as  follows: 


1.  The  model  simulated  aircraft  operations  covering  a 
100-day  period  for  a  base  that  had  20  ope  rational  aircraft. 

2.  The  simulated  base  was  assigned  562  maintenance 
technicians  who  performed  scheduled  and  unscheduled  mainte¬ 
nance  both  on  aircraft  and  .'n  the  shop. 


3.  Twenty  systems  were  defined  on  the  aircraft,  with 
each  system  having  some  number  of  components. 


4.  The  component  failure  rate  and  repair  characteristics 
werq  determined  from  66-1  historical  data,  generated  from 
C-141  maintenance  history. 


5.  During  simulation,  434  missions  were  flown.  The 
mission  demand  and  characteristics  were  determined  by  a 
Monte  Carlo  technique  and  were  based  upon  historical  C-141 
data  taken  from  the  F-l  forms. 


6.  The  probability  of  a  maintenance  error,  by  designated 
type,  in  the  system,  was  determined  from  a  detailed  analysis 
of  the  66-1  data. 


7.  Other  input  data,  such  as  schedule  inspection,  charac¬ 
teristics,  pre-  and  postflight  characteristics,  abort  rate,  and 
shop  procedures,  were  determined  from  published  Air  Force 
documents,  field  interviews,  and. field  observations. 


Two  approaches  to  the  analysis  of  the  effect  of  PIMO  upon  the  C-141A 
system  characteristics  were  taken.  The  first  approach  was  to  deter- 


mine  the  impact  upon  operational  characteristics  of  the  inclusion 
of  PIMO  !nto  all  on-aircraft  unscheduled  maintenance. 

The  second  approach  was  to  determine  the  optimum  reduction  in 
maintenance  personnel  possible  when  PIMO  was  incorporated  into 
all  unscheduled  maintenance  --  with  the  proviso  that  the  operational 
characteristics  would  be  no  worse  than  are  currently  experienced. 

The  operational  characteristics  chosen  for  study  were:  total  number 
of  mission  delays,  total  aircraft  hours  and  maintenance  function,  and 
percent  of  time  the  aircraft  were  operationally  ready.  The  reader 
should  be  cautioned,  that  the  AMES  model  is  a  mathematical  repre¬ 
sentation  of  the  real  system,  and  as  such  it  was  designed  to  be  highly 
representative  of  that  system  and  sensitive  to  the  3ame  effects  that 
occur  in  that  system.  However,  it  was  not  designed  to  be  an  exact 
representation  of  the  real  system.  In  fact,  it  would  be  almost  im¬ 
possible  to  develop  a  simulation  model  which  would  be  exhaustibly 
congruent  to  anything  as  complex  as  aircraft  operations  on  a  given  Air 
Force  base.  Therefore,  it  should  be  remembered  that  some  of  the 
simulation  outputs  --  particularly  those  developed  as  a  nominal  case  -- 
may  differ  somewhat  from  those  actually  observed  in  the  field.  This 
does  not,  however,  invalidate  the  results.  On  the  contrary,  it  indicates 
that  the  AMES  model  results  should  be  viewed  in  terms  of  their  relative 
merits,  rather  than  their  absolute  merits. 

Exercise  of  the  AMES  model  itself  consisted  of  a  series  of  simulation 
runs.  Each  run  consisted  of  20  aircraft  over  a  100-day  period.  Deri¬ 
vation  of  the  nominal  case  from  the  AMES  model  consisted  of  3  runs, 
with  the  only  change  that  of  a  random-feed  input  to  the  model.  Three 
runs  were  made  in  order  to  reduce  the  run-to-run  variance.  The 
average  of  the  three  runs  was  then  used  to  determine  the  nominal 
performance. 

The  exercise  of  the  AMES  model  with  the  PIMO  data  base,  consisted 
of  a  series  of  runs  with  those  previously  stated  modifications  in  the 
input  data  bank. 
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Results  of  AMES  Model  Run 


Figure  6-17  shows  the  effect  of  PIMO  implementation  on  the  number 
of  mission  delays  parameter.  For  purposes  of  the  model  runs,  a 
mission  delay  was  defined  as  any  mission  that  was  unable  to  depart 
within  3-1/2  hours  from  the  time  of  the  initial  mission  demand.  Since 
the  model  is  so  designed  that  the  preflight  inspection  does  not  begin 
until  a  mission  is  demanded,  the  3-1/2  hours  normal  delay  between 
mission  demand  and  a  mission  departure  was  felt  to  be  comparable 
to  actual  operations.  It  should  be  emphasized  that  the  mission  delays 
as  identified  by  the  AMES  model  were  only- those  which  were  mainte¬ 
nance  related.  Primarily,  this  is  because  it  seemed  relevant  and 
because  the  model  was  unable  to  simulate  delays  due  to  other  causes, 
such  as  cargo,  flight  crews,  weather,  etc. 

Within  the  AMES  model,  an  aircraft  may  be  delayed  due  to  any  mechani¬ 
cal  failure  regardless  of  its  magnitude.  In  actual  operations,  on  the 
other  hand,  only  certain  equipment  failures  are  considered  crucial 
enough  to  delay  an  aircraft  mission.  If  the  malfunction  is  minor,  often 
the  aircraft  will  depart  with  the  existing  malfunction.  From  this,  it 
can  be  seen  that  the  number  of  departure  delays,  as  a  consequence  of 
the  AMES  model  run,  will  be  greater  in  most  instances  than  those  ac¬ 
tually  experienced  in  the  field . 

Inspection  of  Figure  6-17  shows  that  if  PIMO's  ability  to  reduce  error 
ra^e  is  disregarded  (0%  reduction  in  errors),  the  benefits  of  low-skilled 
personnel  being  able  to  perform  skilled  maintenance  produces  sufficiently 
fewer  delays  to  more  than  offset  the  additional  time -in-function  required 
to  perform  such  maintenance.  Then,  as  the  maintenance  error  rate  is 
reduced,  significant  improvements  in  over-all  system  performance  is 
obtained.  For  example,  at  the  50%  reduction  in  troubleshooting,  cor¬ 
rective  action  and  operational  check  errors,  the  AMES  model  indicates 
that  mission  delays  would  be  reduced  by  approximately  50%.  As  an  aside, 
the  PIMO  field  test  indicates  that  the  PIMO  Job  Guides  will  reduce  mainte¬ 
nance  errors  by  significantly  more  than  50%.  In  fact,  no  maintenance 


errors  were  observed  when  technicians  used  PIMO.  This  is  not  to 
say  that  no  errors  would  occur  in  a  non-test  situation;  therefore,  if 
these  data  were  discounted,  one  could  reasonably  expect  between 
80-90%  reduction  in  errors.  At  the  75%  level  of  reduction  in  mainte¬ 
nance  errors,  the  number  of  mission  delays  is  reduced  by  approxi¬ 
mately  65%.  At  this  point  the  curve  has  begun  to  level  out,  and  when 
it  reaches  the  100%  level  of  error  rate  reduction,  the  number  of  mis¬ 
sion  delays  is  still  reduced  by  approximately  65%. 

2.  Impact  on  Unscheduled  Maintenance 

The  effect  of  PIMO  on  total  aircraft  hours  and  unscheduled  maintenance 
is  shown  in  Figure  6-18.  This  figure  reflects  only  the  time  in  unsched¬ 
uled  maintenance  at  the  home  base.  Moreover  these  figures  do  not 
include  preflight,  postflight,  scheduled  inspection  times,  or  time  for 
TCTO  changes.  They  dc  include  the  time  the  aircraft  are  in  NORS. 

The  same  conclusions  may  be  drawn  from  this  curve  as  were  drawn 
from  the  previous  curve,  almost  without  exception.  For  a  50%  re¬ 
duction  in  maintenance  errors,  the  total  aircraft  hours  in  unscheduled 
maintenance  is  reduced  by  37%.  This  continues  until  the  100%  level  of 
reduction  in  errors  is  reached,  where  a  maximum  of  44%  improvement 
in  unscheduled  maintenance  is  seen. 

The  consequence  of  this  effect  is  such  that  one  can  see  that  there  would 
be  necessarily  fewer  maintenance  technicians  required  or  fewer  parts 
stores  required,  or  both.  In  addition,  since  the  aircraft  would  be  avail¬ 
able  more  hours  for  operational  mission  assignment,  a  reduction  in  life 
cycle  costs  for  the  aircraft  would  be  realized. 

3.  Percent  of  Time  Operationally  Ready 

Figure  6-19  shows  the  effect  of  PIMO  on  the  operational  readiness 
state  of  the  aircraft.  These  figures  reflect  both  the:  time  away  from 
home  base  and  the  time  at  the  home  base,  for  which  the  aircraft  are 
designated  as  operationally  ready.  Operational  readiness  is  defined 


as  a  state  other  than  that  of  preflight,  postflight,  scheduled,  unsched¬ 
uled  maintenance  or  NORS.  Inspection  of  this  curve  shows  that  when 
maintenance  errors  are  reduced  50%,  they  were  followed  by  a  reduc¬ 
tion  in  non-operational  readiness  state  of  38%.  By  a  75%  reduction  in 
errors,  the  improvement  in  non-operational  readiness  has  nearly 
reached  its  asymptote  --  since  for  a  100%  error  reduction  the  improve¬ 
ment  in  reduction  of  non-operational  readiness  is  40%.  It  is  important 
to  realize  that  this  improvement  comes  not  so  much  from  a  reduction 
in  erroi  rate  as  it  does  from  the  fact  that  low  skilled  personnel  have 
been  included  in  the  pool  of  ava 'able  maintenance  technicians.  Use 
of  people  at  this  experience /training  level  means  that  there  were  less 
queues  in  maintenance,  produced  by  the  lack  of  maintenance  personnel. 

4.  Analysis  of  Maintenance  Personnel  Requirements 

Another  tack  taken  with  the  AMES  model  was  to  reduce  the  maintenance 
personnel  until  the  model  showed  a  performance  for  the  operational 
characteristics,  using  PIMG,  equal  to  that  of  the  nominal  case.  At 
this  point,  a  conclusion  could  be  made  that  if  maintenance  technicians 
did  employ  the  PIMO  formatted  data  the  number  of  personnel  could  be 
reduced  by  some  amount,  with  the  operational  characteristics  equal 
to  those  currently  in  existence.  This  does  not  mean  that  one  could 
hypothesize  a  personnel  reduction  across  ail  areas  or  all  skills,  but 
rather  only  those  which  PIMO  would  effect.  For  example,  personnel 
reduction  would  not  apply  to  scheduled  inspection,  support  shops, 
transistory  aircraft  personnel,  or  supervisory  personnel.  Figure 
6-20  represents  the  upshot  of  the  model  runs  using  the  number  of 
mission  delays  as  the  criteria.  It  can  be  seen  that  if  PIMO  were  to 
reduce  the  error  rate  by  50%,  personnel  could  be  reduced  by  30%  with 
no  penalty  being  paid  through  generating  more  mission  delay  3  than  are 
currently  experienced.  In  a  like  fashion,  when  a  75%  reduction  in 
errors  is  hypothesized,  a  36%  reduction  in  personnel  becomes  possible. 
Finally,  for  a  100%  reduction  in  errors,  one  could  realize  a  30%  per¬ 
sonnel  reduction.  Here  again  while  a  100%  reduction  m  errors  was 
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Figure  6-20  Effect  of  Performance  Improvement  on  Departure  Reliability 

and  Personnel  Reduction 


observed  in  the  field  test  it  is  unwise  to  expect  that  circumstance  to 
continue  in  a  real-life  situation.  Nonetheless,  even  with  a  50%  re¬ 
duction  in  error  approximately  1/3  of  the  personnel  could  be  reduced 
with  no  increase  in  mission  delays  than  are  currently  in  effect.  This 
has  a  dramatic  impact  on  both  over-all  system  life-cycle  costs,  and 
on  the  quality  of  maintenance  possible  in  some  locations  of  the  world 
where  complete  maintenance  staffing,  as  currently  visualized,  would 
not  be  possible. 

Figure  8-21  shows  a  similar  curve  for  the  total  aircraft  hours  in  un¬ 
scheduled  maintenance.  Here,  one  can  see  that  the  results  are  only 
slightly  less  favorable  than  from  that  curve  which  represents  the  ef¬ 
fect  on  mission  delays.  Once  again,  if  one  were  to  hypothesize  a  75% 
reduction  in  errors,  the  model  run  indicates  that  one  could  reduce  per¬ 
sonnel  by  34%  with  no  penalty  of  increase  in  total  aircraft  hours  or 
unscheduled  maintenance. 

Finally,  Figure  6-22  supports  the  results  and  conclusions  drawn  from 
the  two  previous  curves.  Again  taking  the  75%  reduction  in  errors  as 
a  reasonable  departure  from  the  observed  error  reduction  rate  see& 
in  the  field  test,  the  mode]  indicates  that  a  37%  reduction  in  personnel 
could  be  effected  with  no  penalty. 

The  obvious  conclusion  to  be  drawn  is  that  through  the  use  of  PIMO 
formatted  job  guides,  a  reduction  both  in  staff  and  in  maintenance 
errors  could  be  realized.  This  would  then  reflect  significant  savings, 
both  on  the  resources  of  personnel  and  parts  stores.  In  addition,  one 
can  readily  see  that  when  aircraft  are  unavailable  for  mission  assign¬ 
ment,--  due  to  maintenance  requirements  --  the  costs  for  the  aircraft 
system  continue  to  soar,  since  flight  crews  are  not  productive. 

These  runs,  while  derived  from  simulation,  do  have  validity  and  are 
indirect  consonance  with  the  results  of  the  basic  and  the  special  study. 
This  is  not  surprising,  since  the  differences  between  a  nominal  case 
and  the  effect  of  PIMO  on  the  parameters  of  study,  were  based  on  data 
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Figure  6-21.  Effect  of  PIMO,  With  a  Reduction  of  Personnel 
Upon  Aircraft  Hours  in  Maintenance 
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taken  from  the  field  study.  Since  some  of  the  earlier  analyses  have 
demonstrated  that  the  mean  performance  times  observed  during  the 
field  test  may  be  inaccurate  and  not  reflect  the  long  term  expected 
benefits  (because  they  include  the  early  portions  of  the  learning  curve) 
it  was  decided  to  exercise  the  AMES  model  again.  This  time  it  was 
based  on  the  performance  times  related  to  the  toe  of  the  learning  curve 
as  opposed  to  the  over-all  means.  During  the  field  tests,  for  example, 
it  was  generally  found  that  it  took  longer  to  perform  maintenance  tasks 
using  PIMO,  than  to  perform  the  same  task  under  SOP.  Since  the 
learning  curves  demonstrated  that  technicians  --  through  continued  use 
of  PIMO  materials  in  the  same  job  --  were  able  to  significantly  reduce 
their  performance  time,  until  these  times  approached  those  of  skilled 
technicians  using  SOP,  it  was  decided  to  use  those  values  as  the  new 
data  pool.  If  this  hypothesis  can  be  accepted  --  that  the  expected  long- 
run  PIMO  times  are  equal  to,  or  less  than,  the  current  methods  of 
operation  --  then  the  results  generated  by  the  previous  AMES  model 
runs  are  conservative.  For  purposes  of  these  later  runs  it  was  hy¬ 
pothesized  that  the  long  run  PIMO  time  was  equal  to  the  current  SOP 
times.  In  this  later  analysis,  the  nominal  case  remained  identical  to 
that  described  for  the  runs  which  were  presented  earlier. 

Five  model  runs  were  made  of  the  PIMO  case,  each  at  a  different  error 
rate,  0,  25,  50,  75,  and  100%  reduction  in  errors.  The  run3  were  con¬ 
ducted  in  such  a  way  as  to  test  only  the  effect  upon  the  operational 
characteristics,  and  not  the  effects  upon  a  personnel  reduction. 

The  results  of  these  model  runs  are  presented  in  Figures  6-23,  6-24, 
and  6-25.  In  these  figuies,  one  will  see  the  curves  generated  as  a 
result  of  the  second  AMES  run,  as  contrasted  with  the  curve  generated 
during  the  first  run.  These  series  of  curves  indicate  that  as  inexperi¬ 
enced  technicians  become  skilled  in  the  use  of  PIMO  and  skilled  in  the 
maintenance  of  the  aircraft,  the  benefits  will  increase  significantly 
over  those  reported  in  the  earlier  study.  In  all  cases,  the  new  curve 
is  found  to  be  somewhat  parallel  to  the  original  curve,  but  in  the 
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PIMO  (OBSERVED  PIMO  TIMES) 


direction  of  greater  effectiveness.  The  same  conclusions  drawn  from 
the  earlier  set  of  curves  can  be  applied  to  this  set,  but  with  the  degree 
of  systems -effectiveness  enhancement  shif*ed  in  the  positive  direction. 
Finally,  one  should  be  cautious  about  these  particular  sets  of  curves 
since  they  represent  hypothesised  values,  as  opposed  to  observed 
values. 

F.  SUMMARY  CONCLUSIONS 

Section  IV  of  this  volume  listed  fiv*r  specific  hypotheses  that  had  been, 
generated  as  a  consequence  of  earlier  studies.  These  hypotheses  will 
be  discussed  and  analyzed  individually. 

1.  The  originally  intended  criterion  measure  for  this  hypothesis 
was  the  mean  performance  time  for  all  personnel  with  PIMO  data  and 
using  the  SOP.  On  the  basis  of  the  test,  this  hypothesis  must  be  re¬ 
jected.  However,  a  major  finding  of  the  test  was  that  inexperienced 
personnel  can  perform  using  PIMO  data.  Inexperienced  personnel 
constitute  a  significant  fraction  of  the  total  maintenance  staff  at  any 
base.  Effective  use  of  these  individuals  is  more  significant  to  over¬ 
all  system  efficiency  than  a  10%  reduction  in  maintenance  time.  If 
the  hypothesis  is  considered  to  cover  inexperienced  technicians  only 
using  PIMO  data  versus  using  the  SOP,  the  hypothesis  is  accepted. 


Also,  if  one  uses  system -wide  maintenance  time  as  the  reference, 
the  hypothesis  can  be  accepted  marginally  since  there  is  a  decrease 
in  an  aircraft  being  out  of  service  due  to  maintenance  requirements. 

2.  The  second  hypothesis  stated:  PIMO  data  presented  by  the  audio¬ 
visual  mode  would  result  in  an  additional  10%  redaction  in  maintenance 
man-hours.  Here,  there  are  no  data  to  support  this  hypothesis.  In 
fact,  it  would  seem  that  in  most  cases  there  were  no  statistical  sig¬ 
nificances  associated  with  the  difference  in  performance  times  be¬ 
tween  audio-visual  and  booklet  presentation  mode.  Therefore,  in  all 
scientific  candor  it  must  be  concluded  that  this  hypothesis  was  not 
supported. 


Vol.2 


6-7S 


3.  The  third  hypothesis  stated:  a  45%  reduction  in  maintenance 
errors  would  be  possible  when  technicians  used  the  PIMO  Job  Guide. 

This  hypothesis  can  be  accepted  at  the  most  stringent  level  of  signifi¬ 
cance.  This  is  true  because  there  were  jno  observed  maintenance  er¬ 
rors  when  the  technicians  were  using  the  PIMO  Job  Guides.'  This 
represents  not  a  45%,  but  indeed  a  100%  reduction  in  maintenance  er  ¬ 
rors. 

4.  The  fourth  hypothesis  stated:  for  80%  of  the  maintenance  tasks 
that  might  occur,  a  3-level  technician  with  PIMO  could  perform  as 
well  as  a  5-level  technician  with  standard  USAF  Technical  Orders. 

Here  again  there  is  clear  evidence  in  support  for  this  hypothesis.  In 
fact,  the  data  tend  to  suggest  that  not  80%,  but  100%  of  the  maintenance 
tasks  could  be  performed  by  3 -level  technicians  as  well  as,  or  better 
than,  the  5 -level  technician  with  the  standard  USAF  Technical  Orders. 
This  was  brought  out  most  dramatically  in  the  special  test,  where  the 
maintenance  activities  were  contrasted  directly  between  the  use  of  PIMO 
and  the  use  of  the  standard  technical  orders.  It  will  be  recalled,  that 

in  those  cases  where  an  inexperienced  and  naive  technician  was  asked 
to  perform  maintenance  with  tech  orders,  in  some  cases  he  was  un¬ 
able  to  do  so.  Where  he  had  had  previous  experience  on  the  PIMO 
Job  Guides  --  and  subsequently  was  asked  to  perform  maintenance 
using  the  tech  orders  --  his  level  of  experience  was  such  that  he  was 
able  to  complete  it  with  a  minimum  of  errors. 

5.  The  last  hypothesis  stated  that  a  significant  reduction  in  system- 
wide  spares  consumption  would  result.  The  empirical  data  to  support 
this  hypothesis  would  come  from  the  TSA  effectiveness  study,  which 
clearly  indicated  that  there  were  indeed  fewer  troubleshooting  errors 
committed  with  the  TSA  manuals.  The  AMES  model  tended  to  support 
this  hypothesis  through  simulation  rather  than  through  empirical  evi¬ 
dence.  (Nonetheless,  the  reliability  and  validity  of  the  AMES  model 

is  great  enough  to  lead  one  to  accept  this  hypothesis).  The  AMES 
model  did  state  that  there  would  be  a  significant  reduction  in  mainte¬ 
nance  errors  as  a  consequence  of  PIMO,  and  it  would  seem  logical  to 
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extrapolate  this  to  its  orderly  conclusion,  which  would  say  that  since 
better  maintenance  was  performed,  fewer  spares  would  be  consumed. 

More  generally,  the  three  immediate  behavioral  effects  which  were 
expected  were  realized.  There  were  real  and  direct  reductions  in 
maintenance  errors,  at  least  an  indirect  reduction  in  maintenance 
time  --  depending  on  what  is  used  as  a  reference  --  and  clear  evidence 
to  support  the  hypothesis  that  the  ability  of  apprentice  technicians  to 
perform  specialized  maintenance  with  use  of  PIMO  is  adequate. 


SECTION  VII 


AIRCRAFT  MAINTENANCE  EFFECTIVE  SIMULATION  (AMES) 

A.  INTRODUCTION 

It  was  recognized  early  in  the  PIMO  Project  that  in  order  to  make  the 
benefits  of  PIMO  meaningful  they  had  to  be  expressed  in  terms  of  a 
given  system.  In  this  case,  the  C-141  aircraft  was  named  as  the  test 
vehicle.  In  order  for  the  value  of  the  maintenance  system  to  be  fully 
demonstrated,  it  was  necessary  that  this  evaluation  be  couched  in 
objective  and  quantified  terms;  therefore,  a  means  was  devised  which 
would  relate  changes  ii.  maintenance  performances  to  changes  in  the 
over-all  C-141A  system  effectiveness.  The  means  developed  anri 
tested  was  the  AMES  Model.  AMES,  an  acronym  for  Aircraft 
Maintenance  and  Effectiveness  Simulation,  is  a  digital  simulation 
model  programmed  for  the  IBM  7094.  This  model  is  a  technique 
which  allows  the  representation  of  a  real  system  by  means  of  a  dynamic 
mathematical  analog  of  that  system.  This  analog  is  not  identical  to 
the  real  system  but  has  a  high  enough  degree  of  similarity  to  Reflect 
th*.  interactions  and  changes  as  might  be  expected  by  the  real  system. 

This  model  was  designed  by  Serendipity  and  employed  to  evaluate: 

1)  effectiveness  of  aircraft  maintenance,  and  2)  a  means  of  improving 
aircraft  maintenance.  It  is  generally  accepted  that  the  major  advan¬ 
tage  of  any  simulation  model  is  the  ability  of  that  model  to  test 
hypotheses  without  having  to  exercise  the  real  system.  Obviously, 
an  exercise  of  the  real  system  would  result  in  operating  expenses 
which  might  far  exceed  the  value  to  be  gained  by  the  results  of  that 
exercise.  In  fact,  this  very  attribute  of  expense  associated  with 
exercise  of  a  real  system  often  tends  to  inhibit  what  might  prove  to 
be  productive  research.  Nonetheless,  any  model  is  limited  by  the 
extent  to  which  it  responds  as  a  fair  representation  of  its  real  world 
system.  One  must  be  cautious  not  to  exercise  the  model  beyond  the 
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constraints  included  during  its  development.  Lastly,  one  should  be 
cautious  and  use  judgment  in  the  evaluation  of  any  of  the  results  genera¬ 
ted  by  a  model  run. 

B.  STRUCTURE  OF  AMES 

1.  Basic  Logic 

The  AMES  Model  basically  is  a  reflection  of  the  system  diagram  in 
Figure  7-1.  As  illustrated,  each  of  the  functions  are  represented  by 
subroutines  which  determine:  1)  that  the  function  can  be  initiated  using 
available  resources,  2)  the  time  required  to  perform  the  function,  and 
3)  the  effect  of  maintenance  errors  on  system  effectiveness.  Included 
also  are  supervisory  routines  which  control  the  movement  of  aircraft 
through  the  function  which  monitors  and  maintains  a  resource  level 
such  as  spares,  personnel  and  AGE  equipment  and  finally  manages  the 
necessary  inputs  such  as  mission  demands.  The  basic  structure  of  the 
model  is  logically  consistent  with  observed  C-141A  operations. 

2.  Basic  Operations 

The  model  is  so  configured  that  it  can  accept  a  squadron  of  twenty 
aircraft  simultaneously.  Aircraft  status  such  as  operational  ready 
state  is  determined  from  its  apparent  condition  upon  entering  each 
function.  Previously  supplied  input  data  to  the  model  determines  the 
personnel  requhed  to  perform  any  given  function  on  the  system.  The 
capacity  for  the  model  allows  for  a  maximum  of  ten  different  types  or 
skill  levels  to  be  identified.  Delays  in  maintenance  for  a  specific 
technical  specialty  may  occur  in  a  model  run  even  though  a  lesser 
experienced  personnel  of  the  same  specialty  may  be  available.  This 
is  true  only  as  long  as  the  prestated  assignment  policy  indicates  that 
only  the  qualified  technician  is  permitted  to  perform  that  maintenance 
action. 

The  model  can  accommodate  up  to  twenty  aircraft  systems  with  each 
system  having  a  maximum  of  twenty-nine  components.  Equipment 
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failures  do  occur  randomly  according  to  a  distribution  described  for 
each  component.  Subsequent  to  their  removal,  components  are  traced 
through  shop  or  depot  repair  and  then  on  into  base  supply.  Bad  com¬ 
ponents  resulting  from  erroneous  maintenance  are  shown  to  enter  base 
supply  and  do  have  an  effect  on  the  probability  of  removing  a  bad  item 
for  installation  on  an  aircraft.  Spares  may  be  depleted  resulting  in 
the  aircraft  under  consideration  being  placed  in  a  NORS  status. 

Maintenance  functiqn  errors  are  simulated  by  assigning  two  states  for 
aircraft  and  components  --  the  actual  state  and  the  apparent  state. 
Inherent  failures  are  allowed  to  occur  only  in  the  mission  function,  or 
as  the  result  of  damage  error  in  the  repair  function  ("d"  error).  A 
failure  or  damage  error  establishes  the  actual  state  of  a  system  and 
component  as  bad.  It  is  then  the  duty  of  the  air  crew  to  identify  the 
apparent  state  of  the  system.  When  this  is  accomplished  without  error, 
the  actual  and  apparent  state  are  congruent.  Similar  to  equipment 
failures,  personnel  errors  are  said  to  occur  randomly  according  to 
predescribed  distributions. 

It  is  possible  to  relate  changes  in  maintenance  function  performance 
time  and  reliability  to  system  performance  through  the  simple  expedient 
of  following  aircraft  and  aircraft  components.  An  error  in  trouble¬ 
shooting  may  result  in  a  mission  abort  and  may  ultimately  require 
follow-up  maintenance  action.  Thus,  additional  aircraft  ground  time 
is  accumulated.  Maintenance  function  variables  have  an  indirect 
effect  on  the  real  world  system  in  that  they  interact  with  such  factors 
as  resource  availability,  and  personnel  utilization.  If,  for  example, 
function  time  is  reduced,  fewer  delays  are  then  incurred  as  a  con¬ 
sequence  of  personnel  unavailability.  Moreover,  improvements  in 
function  reliability  reduce  the  demand  for  spare  components. which 
were  erroneously  removed,  thereby  reducing  delays  due  to  unavaila¬ 
bility  of  parts.  Since  the  AMES  Model  simulates  the  interactions  of 
the  availability  of  spares,  personnel  and  AGE  equipment,  it  reflects 
the  effect  of  maintenance  performance  improvements. 
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AMES  MODEL  INPUT  DATA 

The  AMES  program  currently  accepts  data  which  describes  the  aircraft 
in  terms  of  its  major  subsystems  and  their  principal  components. 

Failure  rates,  repair  policies,  spares  levels  and  data  required  for 
scheduled  and  unscheduled  maintenance  must  be  furnished  to  the 
program  as  well.  In  addition,  data  which  specifies  the  required 
numbers  and  types  of  personnel  is  required  as  well  as  task  performance 
times  and  personnel  reliability.  In  this  way,  the  set  of  input  data 
established  the  pool  of  manpower,  the  number  of  shifts  and  the  pool 
of  maintenance  equipment.  External  mission  demands  then  act  to 
drive  the  simulation. 

Initially,  the  model  was  used  to  establish  nominal  values  representative 
of  C-141  aircraft  operations  currently  in  effect.  The  nominal  case 
obviously  did  not  reflect  the  effect  of  PIMO.  This  nominal  case  was 
derived  from  input  gathered  from  existing  Air  Force  published  reports, 
raw  Air  Force  maintenance  and  operational  data  (66-1  and  F-l),  field 
interviews  and  field  observations.  Once  the  nominal  case  was  estab¬ 
lished,  the  AMES  model  was  then  used  to  determine  the  effect  of  PIMO 
on  over-all  systems  operation  and  maintenance.  This  effect  is  measured 
or  demonstrated  quantitatively  through  its  impact  on  the  variance  of  the 
specific  parameters  mentioned  earlier,  i.e.,  mission  readiness. 
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13  ABSTRACT 


This  report  describes  the  latest  phase  in  the  program  to  develop  and 
evaluate  PIMO  (Presentation  of  Information  for  Maintenance  and  Operation); 
a  job  guide  concept  applied  to  maintenance.  Between  August  1968  and  April 
1868,  a  test  was  conducted  at  Charleston  AFB,  South  Carolina,  to  deter¬ 
mine  the  effectiveness  of  PIMO.  Three  immediate  behavioral  effects  were 
expected:  1)  reduction  in  maintenance  time,  2)  reduction  in  maintenance 
errors,  and  3)  allow  usage  of  inexperienced  technicians  with  no  significant 
penalty.  Experienced  and  inexperienced  Air  Force  technicians  performed 
maintenance  on  C-141A  aircraft  using  PIMO  Job  Guides  presented  in  audio¬ 
visual  and  booklet  modes.  Performance  was  measured  in  terms  of  time  to 
perform  and  procedural  errors.  The  performance  was  compared  with  the 
performance  on  the  same  jobs  by  a  control  group,  i.  e. ,  experienced  tech¬ 
nicians  performing  in  the  normal  manner.  The  following  conclusions  were 
drawn  from  the  test  results:  1)  after  initial  learning  trials,  both  experi¬ 
enced  and  inexperienced  technicians  using  PIMO  can  perform  error-free 
maintenance  within  the  same  time  as  experienced  technicians  performing 
in  the  no-mal  manner,  2)  inexperienced  technicians  perform  as  well  as 
experienced  technicians  when  both  use  PIMO,  3)  there  is  no  significant 
difference  between  audio-visual  and  booklet  modes,  4)  the  users  revealed 
an  overwhelmingly  positive  reaction  to  PIMC,  and  5)  the  performance  im¬ 
provements  provide  the  capabilities  to  significantly  improve  system  per¬ 
formance  defined  in  terms  of  departure  reliability,  time -in-maintenance, 
and  operational  readiness.  This  report  also  presents  a  description  of  the 
recommended  operational  system,  specifications  and  guidelines  for  PIMO 
format  development,  including  troubleshooting. 
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